Surgical methods using multi-source imaging

ABSTRACT

In general, devices, systems, and methods for multi-source imaging are provided.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Prov. Pat. App. No.63/249,644 entitled “Surgical Devices, Systems, And Methods UsingMulti-Source Imaging” filed Sep. 29, 2021, which is hereby incorporatedby reference in its entirety.

FIELD

The present disclosure relates generally to surgical devices, systems,and methods using multi-source imaging.

BACKGROUND

Surgical systems often incorporate an imaging system, which can allowmedical practitioners to view a surgical site and/or one or moreportions thereof on one or more displays, e.g., a monitor, a computertablet screen, etc. The display(s) can be local and/or remote to asurgical theater. The imaging system can include a scope with a camerathat views the surgical site and transmits the view to the one or moredisplays viewable by medical practitioner(s).

Imaging systems can be limited by the information that they are able torecognize and/or convey to the medical practitioner(s). For example,certain concealed structures, physical contours, and/or dimensionswithin a three-dimensional space may be unrecognizable intraoperativelyby certain imaging systems. For another example, certain imaging systemsmay be incapable of communicating and/or conveying certain informationto the medical practitioner(s) intraoperatively.

Accordingly, there remains a need for improved surgical imaging.

SUMMARY

In general, devices, systems, and methods for multi-source imaging areprovided.

In one aspect, a surgical method is provided that in one embodimentincludes visualizing, with a first imaging system, a first side of atissue wall during performance of a surgical procedure. The firstimaging system is a thoracoscopic imaging system or a laparoscopicimaging system. The method also includes visualizing, with an endoscopicimaging system, a second, opposite side of the tissue wall during theperformance of the surgical procedure. The first imaging system and theendoscopic imaging system are unable to directly visualize each other,and the first imaging system and the endoscopic imaging system areoperatively coupled together. The method also includes guiding a firstsurgical instrument based on the visualization provided by the firstimaging system and the visualization provided by the endoscopic imagingsystem, and guiding a second surgical instrument configured based on thevisualization provided by the endoscopic imaging system and based on thevisualization provided by the first imaging system.

The method can vary in any number of ways. For example, at least one ofthe first and second surgical instruments can include integratedtracking and coordinating means that identifies a location of the atleast one first surgical instrument and second surgical instrument, andthe method can further include communicating the location to the firstimaging system and the endoscopic imaging system so as to providesecondary location verification to the first imaging system and theendoscopic imaging system. For another example, the method can furtherinclude determining, with the first imaging system, a location andorientation of the second surgical instrument relative to the firstsurgical instrument to facilitate the first surgical instrument beingguided based on the visualization provided by the endoscopic imagingsystem, and determining, with the endoscopic imaging system, a locationand orientation of the first surgical instrument relative to the secondsurgical instrument to facilitate the second surgical instrument beingguided based on the visualization provided by the first imaging system.For yet another example, the method can further include visualizing,with each of the first imaging system and the endoscopic imaging system,a common anatomic landmark and thereby facilitate guiding of the firstand second surgical instruments relative to one another. For stillanother example, one of the first imaging system and the endoscopicimaging system can include a structured light emitter configured to emita structured light pattern on a surface of an organ, a spectral lightemitter configured to emit spectral light in a plurality of wavelengthscapable of penetrating the organ and reaching an internal aspect locatedbelow the surface of the organ, and an image sensor configured to detectreflected structured light pattern, reflected spectral light, andreflected visible light, and the surgical method can further includeconstructing, with a controller, a three-dimensional (3D) digitalrepresentation of the organ from the reflected structured light patterndetected by the image sensor, detecting, with the controller, theinternal aspect from the reflected spectral light detected by the imagesensor, integrating, with the controller, the internal aspect with the3D digital representation of the organ, and causing, with thecontroller, a display device to show the internal aspect and a 3Drendering of the organ based on the 3D digital representation of theorgan. For another example, the method can further include visualizing,with a computerized tomography (CT) imaging system, the first and secondinstruments and thereby facilitate guiding of the first and secondsurgical instruments relative to one another. For still another example,the method can further include gathering, with a non-magnetic sensingsystem that uses at least one of ultrasonic energy and radiofrequency(RF) energy, data indicative of a location of the first and secondsurgical instruments and thereby facilitate guiding of the first andsecond surgical instruments relative to one another. For anotherexample, the first and second surgical instruments can each bereleasably coupled to a robotic surgical system that controls theguiding of each of the first and second surgical instruments.

In another embodiment, a surgical method includes gathering, with animaging device, images of an organ during performance of a surgicalprocedure. The images include a plurality of images each gathered at adifferent time during the performance of the surgical procedure as theorgan undergoes distortions. The method also includes tracking, with acontroller, the organ from a first state to a second, different statethat is subsequent to the first state by analyzing the plurality ofimages. The tracking allows the controller to predict an internal aspectof the organ in the second state. The method also includes causing, withthe controller, a display device to show an indication of the internalaspect of the organ in the second state.

The method can have any number of variations. For example, the trackingcan forecast a perimeter shape of the internal aspect in the secondstate, a location of the internal aspect in the organ in the secondstate, and an orientation of the internal aspect in the organ in thesecond state. For another example, the plurality of images can beanalyzed with respect to at least one of tissue strain and a geometry ofthe organ.

For yet another example, the imaging device can include a flexiblescoping device, the organ can be a lung, the internal aspect can be atumor. The lung in the first state can be expanded, and the lung in thesecond state can be collapsed. The method can further includecontrolling, with the controller, the collapse of the lung to create apredefined shape of the tumor.

For another example, the imaging device can include a structured lightemitter configured to emit a structured light pattern on a surface ofthe organ, a spectral light emitter configured to emit spectral light ina plurality of wavelengths capable of penetrating the organ and reachingthe internal aspect located below the surface of the organ, and an imagesensor configured to detect reflected structured light pattern,reflected spectral light, and reflected visible light, and the methodcan further include constructing, with a controller, a three-dimensional(3D) digital representation of the organ from the reflected structuredlight pattern detected by the image sensor, detecting, with thecontroller, the internal aspect from the reflected spectral lightdetected by the image sensor, integrating, with the controller, theinternal aspect with the 3D digital representation of the organ, andcausing, with the controller, a display device to show the internalaspect and a 3D rendering of the organ based on the 3D digitalrepresentation of the organ.

For still another example, the organ can be a lung, the imaging devicecan include a bronchoscope, the method can further include advancing thebronchoscope distally in an airway of the lung during the performance ofthe surgical procedure, and the method can further include automaticallycausing, with the controller, the bronchoscope to be retractedproximally in the airway of the lung during the performance of thesurgical procedure. The method can further include automaticallycausing, with the controller, the bronchoscope to be retractedproximally based on an analysis of the images indicating thebronchoscope being within a threshold distance of a tissue wall. Themethod can further include automatically causing, with the controller,the bronchoscope to be retracted proximally also based on at least oneof a rate of change of a volume of the lung and a rate of change of anexternal surface of the lung. The method can further includeautomatically causing, with the controller, the bronchoscope to beretracted proximally also based on the rate of change of the volume ofthe lung as indicated by at least one of CT imaging and ventilatorexchange. The imaging device can include a structured light emitter thatemits a structured light pattern on the external surface of the organ,the imaging device can include an image sensor that detects reflectedstructured light pattern, and the method can further includeautomatically causing, with the controller, the bronchoscope to beretracted proximally also based on the rate of change of the externalsurface of the lung as indicated by the reflected structure lightpattern.

For yet another example, a surgical hub can include the controller. Forstill another example, a robotic surgical system can includes thecontroller, and the imaging device can be releasably coupled to andcontrolled by the robotic surgical system.

In another embodiment, a surgical method includes gathering, with afirst imaging device, first images visualizing a surgical target,gathering, with a second imaging device, second images that enhance thevisualization of the surgical target, and causing, with a controller, adisplay device to show a single view of the surgical target that is acooperation of the first and second images. The first imaging device hasa different point of view of the surgical site than the second imagingdevice.

The method can vary in any number of ways. For example, the differentpoint of view of the first imaging device can be a physically differentpoint of view of the surgical site than the second imaging device, andthe surgical target can not be accurately visible by at least one of thefirst and second imaging devices. The method can further includecausing, with the controller, the point of view of the first imagingdevice to change based on the second images gathered by the secondimaging device. The second imaging device can have a point of viewwithin a body lumen, the first imaging device can have a point of viewoutside the body lumen, the second imaging device can have a point ofview from a first side of a tissue wall, the first imaging device canhave a point of view from a second side of the tissue wall that isopposite to the first side of the tissue wall, or the second imagingdevice can have a point of view from a first side of an organ and thefirst imaging device can have a point of view from a second, differentside of the organ.

For another example, the different point of view of the first imagingdevice can be the first imaging device having a different energymodality than the second imaging device. The different energy modalitycan include the second imaging device having ultrasound or CT modalityand the first imaging device having another type of energy modality.

For yet another example, the different point of view of the firstimaging device can be the first imaging device having a physicallydifferent point of view of the surgical site than the second imagingdevice and having a different energy modality than the second imagingdevice. For another example, the first imaging device can be operated bya first user during a surgical procedure in which the first and secondimages are being gathered, the second imaging device can be operated bya second user during the surgical procedure; and a surgical hub caninclude the controller. For still another example, the first and secondimaging devices can each be releasably coupled to and controlled by arobotic surgical system, and the robotic surgical system can include thecontroller. For another example, a robotic surgical system can includethe controller and the display device, and the first and second imagingdevices can each be releasably coupled to and controlled by the roboticsurgical system. For yet another example, the surgical target caninclude a tumor at a lung, and the first imaging device can include aflexible scoping device.

In another embodiment, a surgical method includes gathering, with afirst imaging device, first images of a surgical site during performanceof a surgical procedure, gathering, with a second imaging device, secondimages of the surgical site during the performance of the surgicalprocedure, analyzing, with a controller, the first images and the secondimages to identify and define boundaries of connective soft tissueplanes, relating, with the controller, the identified and definedboundaries of the connective soft tissue planes to an anatomic structureand a role of the tissue, and causing, with the controller, a displaydevice to show information related to the tissue and to show at leastone of the first images and the second images overlaid with theidentified and defined boundaries of the connective soft tissue planesand thereby define a location and orientation of the connective softtissue planes.

The method can have any number of variations. For example, the anatomicstructure and the role of the tissue can include at least one of tissueplanes, tissue composition, tumor location, tumor margin identification,adhesions, vascularization, and tissue fragility. For another example,the information related to the tissue can include at least one of a typeof the tissue, collagen composition of the tissue, organized versusremodeled disorganized fiber orientation of the tissue, viability of thetissue, and health of the tissue. For yet another example, the firstimaging device can include a structured light emitter configured to emita structured light pattern on a surface of the anatomic structure, aspectral light emitter configured to emit spectral light in a pluralityof wavelengths capable of penetrating the anatomic structure andreaching an embedded structure located below the surface of the anatomicstructure, and an image sensor configured to detect reflected structuredlight pattern, reflected spectral light, and reflected visible light,the method can further include constructing, with the controller, athree-dimensional (3D) digital representation of the anatomic structurefrom the reflected structured light pattern detected by the imagesensor, and the controller can use the 3D digital representation inidentifying and defining the boundaries of the connective soft tissueplanes. For still another example, the first imaging device can includea flexible scoping device, and the second imaging device can includes arigid scoping device. For another example, the first imaging device caninclude a bronchoscope, the anatomic structure can include a lung, themethod can further include advancing the bronchoscope into the lung, thetissue can include bronchial tissue, the embedded structure can includea tumor, and the location and orientation of the connective soft tissueplanes can enable identification of tumor location and orientation inthe lung.

For yet another example, the first imaging device can gather the firstimages using a wavelength outside a spectrum of visible light so as toallow visualization of the embedded structure from outside the anatomicstructure, the wavelength outside the spectrum of visible light caninclude an ultrasound wavelength or an infrared wavelength, and thesecond imaging device can gather the second images using a wavelengthwithin the spectrum of visible light so as to allow visualization of thesurface of the anatomic structure. The method can further includedelivering a contrast agent to the anatomic structure, the first imagingdevice can visualize the contrast agent within the anatomic structure,and the second imaging device cannot visualize the contrast agent withinthe anatomic structure.

For still another example, the first and second imaging devices can bereleasably coupled to and controlled by a robotic surgical system, and asurgical hub can include the controller. For another example, a roboticsurgical system can include the controller and the display device, andthe first and second imaging devices can each be releasably coupled toand controlled by the robotic surgical system.

In another embodiment, a surgical method includes visualizing a surgicalsite with a first visualization system, visualizing the surgical sitewith a second visualization system configured to visualize the surgicalsite, and monitoring, with a controller and during the secondvisualization system’s visualization of the surgical site, energydelivery to tissue at the surgical site. The energy is being deliveredto the tissue by an electrode of a surgical instrument positioned in alumen of the first visualization system. The method also includescontrolling, with the controller, energizing of the electrode such thata parameter associated with the tissue does not exceed a predefinedmaximum threshold.

The method can vary in any number of ways. For example, the predefinedmaximum threshold can include a temperature of the tissue, and themethod can further include measuring the temperature of the tissue witha sensor operatively coupled to the controller. For another example, thepredefined maximum threshold can include a temperature of the tissue,the electrode can include an electrode array, and the method can furtherinclude measuring the temperature of the tissue with the electrodearray. For still another example, the predefined maximum threshold caninclude a surface water content of the tissue, and the method canfurther include measuring the surface water content of the tissue with asensor operatively coupled to the controller. For another example, thepredefined maximum threshold can include a refractivity of the tissue,and the method can further include measuring the refractivity of thetissue with at least one of the first and second visualization systems.For yet another example, the first visualization system can bevisualizing within a hollow organ at the surgical site, the tissue canbe tissue within the hollow organ, and the second visualization systemcan be visualizing outside the hollow organ. For another example, theenergy can include one of radiofrequency (RF) energy and microwaveenergy. For yet another example, the tissue can include a tumor, and theenergy can include cold so as to treat the tumor with cyroablation. Forstill another example, a surgical hub can include the controller. Foryet another example, a robotic surgical system can include thecontroller, and the surgical instrument can be configured to bereleasably coupled to and controlled by the robotic surgical system.

In one aspect, a surgical system is provided that in one embodimentincludes a first imaging device configured to gather first images forvisualization of a surgical target, a second imaging device configuredto gather second images that enhance the visualization of the surgicaltarget, and a controller configured to cause a display device to show asingle view of the surgical target that is a cooperation of the firstand second images. The first imaging device has a different point ofview of the surgical site than the second imaging device.

The surgical system can vary in any number of ways. For example, thedifferent point of view of the first imaging device can be a physicallydifferent point of view of the surgical site than the second imagingdevice, and the surgical target can not be accurately visible by atleast one of the first and second imaging devices. The controller can beconfigured to cause the point of view of the first imaging device tochange based on the second images gathered by the second imaging device.The second imaging device can be configured to have a point of viewwithin a body lumen, the first imaging device can be configured to havea point of view outside the body lumen, the second imaging device can beconfigured to have a point of view from a first side of a tissue wall,the first imaging device can be configured to have a point of view froma second side of the tissue wall that is opposite to the first side ofthe tissue wall, or the second imaging device can be configured to havea point of view from a first side of an organ and the first imagingdevice is configured to have a point of view from a second, differentside of the organ.

For another example, the different point of view of the first imagingdevice can be the first imaging device having a different energymodality than the second imaging device. The different energy modalitycan include the second imaging device having ultrasound or CT modalityand the first imaging device having another type of energy modality.

For yet another example, the different point of view of the firstimaging device can be the first imaging device having a physicallydifferent point of view of the surgical site than the second imagingdevice and having a different energy modality than the second imagingdevice. For still another example, the first imaging device can beconfigured to be operated by a first user during a surgical procedure inwhich the first and second images are being gathered, the second imagingdevice can be configured to be operated by a second user during thesurgical procedure, and a surgical hub can include the controller. Foranother example, the system can further include the display device, arobotic surgical system can include the controller and the displaydevice, and the first and second imaging devices can each be configuredto be releasably coupled to and controlled by the robotic surgicalsystem.

In another embodiment, a surgical system includes a first imaging deviceconfigured to gather first images of a surgical site during performanceof a surgical procedure, a second imaging device configured to gathersecond images of the surgical site during the performance of thesurgical procedure, and a controller configured to analyze the firstimages and the second images to identify and define boundaries ofconnective soft tissue planes, relate the identified and definedboundaries of the connective soft tissue planes to an anatomic structureand a role of the tissue, and cause a display device to show informationrelated to the tissue and to show at least one of the first images andthe second images overlaid with the identified and defined boundaries ofthe connective soft tissue planes and thereby define a location andorientation of the connective soft tissue planes.

The system can vary in any number of ways. For example, the anatomicstructure and the role of the tissue can include at least one of tissueplanes, tissue composition, tumor location, tumor margin identification,adhesions, vascularization, and tissue fragility. For another example,the information related to the tissue can include at least one of a typeof the tissue, collagen composition of the tissue, organized versusremodeled disorganized fiber orientation of the tissue, viability of thetissue, and health of the tissue. For yet another example, the firstimaging device can include a structured light emitter configured to emita structured light pattern on a surface of the anatomic structure, aspectral light emitter configured to emit spectral light in a pluralityof wavelengths capable of penetrating the anatomic structure andreaching an embedded structure located below the surface of the anatomicstructure, and an image sensor configured to detect reflected structuredlight pattern, reflected spectral light, and reflected visible light,the controller can be configured to construct a three-dimensional (3D)digital representation of the anatomic structure from the reflectedstructured light pattern detected by the image sensor, and thecontroller can be configured to use the 3D digital representation inidentifying and defining the boundaries of the connective soft tissueplanes. For still another example, the first imaging device can includea flexible scoping device, and the second imaging device can includes arigid scoping device. For yet another example, the anatomic structurecan include a lung, the first imaging device can include a bronchoscopeconfigured to be advanced into the lung, the tissue can includebronchial tissue, the embedded structure can include a tumor, and thelocation and orientation of the connective soft tissue planes can enableidentification of tumor location and orientation in the lung.

For another example, the first imaging device can be configured togather the first images using a wavelength outside a spectrum of visiblelight so as to allow visualization of the embedded structure fromoutside the anatomic structure, the wavelength outside the spectrum ofvisible light can include an ultrasound wavelength or an infraredwavelength, and the second imaging device can be configured to gatherthe second images using a wavelength within the spectrum of visiblelight so as to allow visualization of the surface of the anatomicstructure. The system can further include a contrast agent configured tobe delivered to the anatomic structure, the first imaging device can beconfigured to visualize the contrast agent within the anatomicstructure, and the second imaging device cannot visualize the contrastagent within the anatomic structure.

For yet another example, the first and second imaging devices can eachbe configured to be releasably coupled to and controlled by a roboticsurgical system, and a surgical hub can include the controller. Foranother example, the system can further include the display device, arobotic surgical system can include the controller and the displaydevice, and the first and second imaging devices can each be configuredto be releasably coupled to and controlled by the robotic surgicalsystem.

In another embodiment, a surgical system includes a first imaging systemconfigured to provide visualization of a surgical site including atissue wall. The first imaging system is a thoracoscopic imaging systemor a laparoscopic imaging system. The system also includes a firstsurgical instrument configured to be guided based on the visualizationprovided by the first imaging system, an endoscopic imaging systemconfigured to be advanced to the surgical site transorally andconfigured to provide visualization of the surgical site including thetissue wall, and a second surgical instrument configured to be guidedbased on the visualization provided by the endoscopic imaging system.With the first imaging system and the endoscopic imaging system beingunable to directly visualize each other, the first imaging system isconfigured to visualize a first side of the tissue wall duringperformance of a surgical procedure and the endoscopic imaging system isconfigured to visualize a second, opposite side of the tissue wallduring the performance of the surgical procedure. The first imagingsystem and the endoscopic imaging system are configured to beoperatively coupled together such that the first surgical instrument isalso configured to be guided based on the visualization provided by theendoscopic imaging system and the second surgical instrument is alsoconfigured to be guided based on the visualization provided by the firstimaging system.

The system can vary in any number of ways. For example, at least one ofthe first and second surgical instruments can include integratedtracking and coordinating means configured to identify a location of theat least one first surgical instrument and second surgical instrumentand can be configured to communicate the location to the first imagingsystem and the endoscopic imaging system so as to provide secondarylocation verification to the first imaging system and the endoscopicimaging system. For another example, the first imaging system can beconfigured to determine a location and orientation of the secondsurgical instrument relative to the first surgical instrument tofacilitate the first surgical instrument being guided based on thevisualization provided by the endoscopic imaging system, and theendoscopic imaging system can be configured to determine a location andorientation of the first surgical instrument relative to the secondsurgical instrument to facilitate the second surgical instrument beingguided based on the visualization provided by the first imaging system.For yet another example, the first imaging system and the endoscopicimaging system can be configured to each visualize a common anatomiclandmark and thereby facilitate guiding of the first and second surgicalinstruments relative to one another. For still another example, one ofthe first imaging system and the endoscopic imaging system includes astructured light emitter configured to emit a structured light patternon a surface of an organ, a spectral light emitter configured to emitspectral light in a plurality of wavelengths capable of penetrating theorgan and reaching an internal aspect located below the surface of theorgan, and an image sensor configured to detect reflected structuredlight pattern, reflected spectral light, and reflected visible light,and the surgical system further includes a controller configured toconstruct a three-dimensional (3D) digital representation of the organfrom the reflected structured light pattern detected by the imagesensor, detect the internal aspect from the reflected spectral lightdetected by the image sensor, integrate the internal aspect with the 3Ddigital representation of the organ, and cause a display device to showthe internal aspect and a 3D rendering of the organ based on the 3Ddigital representation of the organ. For another example, the system canfurther include a computerized tomography (CT) imaging system configuredto provide visualization of the first and second instruments and therebyfacilitate guiding of the first and second surgical instruments relativeto one another. For yet another example, the system can further includea non-magnetic sensing system that uses at least one of ultrasonicenergy and radiofrequency (RF) energy and is configured to gather dataindicative of a location of the first and second surgical instrumentsand thereby facilitate guiding of the first and second surgicalinstruments relative to one another. For another example, the first andsecond surgical instruments can each be configured to be releasablycoupled to a robotic surgical system configured to control the guidingof each of the first and second surgical instruments.

In another embodiment, a surgical system includes an imaging deviceconfigured to gather images of an organ during performance of a surgicalprocedure. The images include a plurality of images each gathered at adifferent time during the performance of the surgical procedure as theorgan undergoes distortions. The system also includes a controllerconfigured to track the organ from a first state to a second, differentstate that is subsequent to the first state by analyzing the pluralityof images. The tracking allows the controller to predict an internalaspect of the organ in the second state. The controller is alsoconfigured to cause a display device to show an indication of theinternal aspect of the organ in the second state.

The system can have any number of variations. For example, the trackingcan forecast a perimeter shape of the internal aspect in the secondstate, a location of the internal aspect in the organ in the secondstate, and an orientation of the internal aspect in the organ in thesecond state. For another example, the plurality of images can beanalyzed with respect to at least one of tissue strain and a geometry ofthe organ.

For yet another example, the imaging device can include a flexiblescoping device, the organ can be a lung, and the internal aspect can bea tumor. The lung in the first state can be expanded, and the lung inthe second state can be collapsed. The controller can be configured tocontrol the collapse of the lung to create a predefined shape of thetumor.

For still another example, the imaging device can include a structuredlight emitter configured to emit a structured light pattern on a surfaceof the organ, a spectral light emitter configured to emit spectral lightin a plurality of wavelengths capable of penetrating the organ andreaching the internal aspect located below the surface of the organ, andan image sensor configured to detect reflected structured light pattern,reflected spectral light, and reflected visible light, and thecontroller can be configured to construct a three-dimensional (3D)digital representation of the organ from the reflected structured lightpattern detected by the image sensor, detect the internal aspect fromthe reflected spectral light detected by the image sensor, integrate theinternal aspect with the 3D digital representation of the organ, andcause a display device to show the internal aspect and a 3D rendering ofthe organ based on the 3D digital representation of the organ.

For yet another example, the organ can be a lung, the imaging device caninclude a bronchoscope configured to be advanced distally in an airwayof the lung during the performance of the surgical procedure, and thecontroller can be configured to automatically cause the bronchoscope tobe retracted proximally in the airway of the lung during the performanceof the surgical procedure. The controller can be configured toautomatically cause the bronchoscope to be retracted proximally based onan analysis of the images indicating the bronchoscope being within athreshold distance of a tissue wall. The controller can be configured toautomatically cause the bronchoscope to be retracted proximally alsobased on at least one of a rate of change of a volume of the lung and arate of change of an external surface of the lung. The controller can beconfigured to automatically cause the bronchoscope to be retractedproximally also based on the rate of change of the volume of the lung asindicated by at least one of CT imaging and ventilator exchange. Theimaging device can include a structured light emitter configured to emita structured light pattern on the external surface of the organ, theimaging device can include an image sensor configured to detectreflected structured light pattern, and the controller can be configuredto automatically cause the bronchoscope to be retracted proximally alsobased on the rate of change of the external surface of the lung asindicated by the reflected structure light pattern.

For still another example, a surgical hub can include the controller.For another example, a robotic surgical system can include thecontroller, and the imaging device can be configured to be releasablycoupled to and controlled by the robotic surgical system.

In another embodiment, a surgical system includes a first visualizationsystem configured to visualize a surgical site, and a surgicalinstrument configured to be advanced to the surgical site through alumen of the first visualization system. The surgical includes anelectrode configured to deliver energy to tissue at a surgical site. Thesystem also includes a second visualization system configured tovisualize the surgical site, and a controller configured to beoperatively coupled to the electrosurgical instrument and to the secondvisualization system, monitor the electrode’s energy delivery to thetissue during the delivery and during the second visualization system’svisualization of the surgical site, and control energizing of theelectrode such that a parameter associated with the tissue does notexceed a predefined maximum threshold.

The system can vary in any number of ways. For example, the predefinedmaximum threshold can include a temperature of the tissue, and thesystem can further include a sensor operatively coupled to thecontroller and configured to measure the temperature of the tissue. Foranother example, the predefined maximum threshold includes a temperatureof the tissue, and the electrode can include an electrode arrayconfigured to measure the temperature of the tissue. For yet anotherexample, the predefined maximum threshold can include a surface watercontent of the tissue, and the system can further include a sensoroperatively coupled to the controller and configured to measure thesurface water content of the tissue. For still another example, thepredefined maximum threshold can include a refractivity of the tissue,and at least one of the first and second visualization systems can beconfigured to measure the refractivity of the tissue. For anotherexample, the first visualization system can be configured to visualizewithin a hollow organ at the surgical site, the tissue can be tissuewithin the hollow organ, and the second visualization system can beconfigured to visualize outside the hollow organ. For yet anotherexample, the energy can include one of radiofrequency (RF) energy andmicrowave energy. For still another example, the tissue can include atumor, and the energy can include cold so as to treat the tumor withcyroablation. For another example, a surgical hub can include thecontroller. For still another example, a robotic surgical system caninclude the controller, and the surgical instrument can be configured tobe releasably coupled to and controlled by the robotic surgical system.

BRIEF DESCRIPTION OF DRAWINGS

The present invention is described by way of reference to theaccompanying figures which are as follows:

FIG. 1 is a schematic view of one embodiment of a surgical visualizationsystem;

FIG. 2 is a schematic view of triangularization between a surgicaldevice, an imaging device, and a critical structure of FIG. 1 ;

FIG. 3 is a schematic view of another embodiment of a surgicalvisualization system;

FIG. 4 is a schematic view of one embodiment of a control system for asurgical visualization system;

FIG. 5 is a schematic view of one embodiment of a control circuit of acontrol system for a surgical visualization system;

FIG. 6 is a schematic view of one embodiment of a combinational logiccircuit of a surgical visualization system;

FIG. 7 is a schematic view of one embodiment of a sequential logiccircuit of a surgical visualization system;

FIG. 8 is a schematic view of yet another embodiment of a surgicalvisualization system;

FIG. 9 is a schematic view of another embodiment of a control system fora surgical visualization system;

FIG. 10 is a graph showing wavelength versus absorption coefficient forvarious biological materials;

FIG. 11 is a schematic view of one embodiment of a spectral emittervisualizing a surgical site;

FIG. 12 is a graph depicting illustrative hyperspectral identifyingsignatures to differentiate a ureter from obscurants;

FIG. 13 is a graph depicting illustrative hyperspectral identifyingsignatures to differentiate an artery from obscurants;

FIG. 14 is a graph depicting illustrative hyperspectral identifyingsignatures to differentiate a nerve from obscurants;

FIG. 15 is a schematic view of one embodiment of a near infrared (NIR)time-of-flight measurement system being utilized intraoperatively;

FIG. 16 shows a time-of-flight timing diagram for the system of FIG. 15;

FIG. 17 is a schematic view of another embodiment of a near infrared(NIR) time-of-flight measurement system being utilized intraoperatively;

FIG. 18 is a schematic view of one embodiment of a computer-implementedinteractive surgical system;

FIG. 19 is a schematic view of one embodiment a surgical system beingused to perform a surgical procedure in an operating room;

FIG. 20 is a schematic view of one embodiment of a surgical systemincluding a smart surgical instrument and a surgical hub;

FIG. 21 is a flowchart showing a method of controlling the smartsurgical instrument of FIG. 20 ;

FIG. 22 is a schematic view of a colon illustrating major resections ofthe colon;

FIG. 22A is a perspective partial cross-sectional view of one embodimentof a duodenal mucosal resurfacing procedure;

FIG. 23 is a schematic view of one embodiment of a CT scan of a patientand of an MRI of the patient;

FIG. 24 is a schematic view of a collapsed lung of a patient having abronchoscope advanced therein;

FIG. 25 is a schematic view of retraction of the bronchoscope of FIG. 24;

FIG. 26 is a graph showing a rate of retraction of the bronchoscope ofFIG. 24 , lung volume reduction, and distance between the bronchoscopeand an intersecting tissue wall versus time;

FIG. 27 is a schematic view illustrating the distance of FIG. 26 ;

FIG. 28 is a schematic view illustrating the lung of FIG. 24 and thelung in an inflated state;

FIG. 29 is a schematic view of a tumor in the lung of FIG. 28corresponding to the inflated and collapsed states of the lung;

FIG. 30 is a graph showing lung collapse percentage versus tumor size inX, Y, and X dimensions;

FIG. 31 is a schematic view of an optimal percentage of lung collapse ascompared to an inflated lung and a fully delated lung;

FIG. 32 is a schematic view of a lung in an inflated state, a partiallycollapses state, and a fully collapses state;

FIG. 33 is a schematic view of the lung of FIG. 32 in the inflated statewith physical organ tag markers and a projected light structure thereon;

FIG. 34 is a schematic view of the lung of FIG. 32 in the inflated statewith physical organ tag markers and a projected light structure thereon;

FIG. 35 is a flowchart of one embodiment of a method of predicting tumorsize using measured tissue strain;

FIG. 36 is a schematic view of a lung inflated and in various states ofcollapse;

FIG. 37 is a schematic view of the lung in two of the collapsed statesof FIG. 36 ;

FIG. 38 shows surface geometry and estimated sub-surface geometry forthe lung of FIG. 36 ;

FIG. 39 is a CT image showing an intersegmental plane between threetissue segments;

FIG. 40 is a CT image showing an intersegmental plane between two of thetissue segments of FIG. 39 ;

FIG. 41 is a schematic view of a display;

FIG. 42 is an enhanced CT image of a lung;

FIG. 43 is a schematic front view of lung airways;

FIG. 44 is a schematic right side view of one embodiment of a path ofadvancement of a scope in a lung;

FIG. 45 is a schematic front view of the path of advancement of thescope of FIG. 44 ;

FIG. 46 is a schematic partially cross-sectional view of the scope ofFIG. 44 in the lung;

FIG. 47 is a flowchart showing one embodiment of a method of usingprimary and secondary imaging systems;

FIG. 48 is a schematic view of one embodiment of a display showing anon-the-fly adaptation of vascular CT with real-time local scanning;

FIG. 49 is a schematic view of first and second imaging systemspositioned relative to a tissue;

FIG. 50 is another schematic view of first and second imaging systemspositioned relative to a tissue;

FIG. 51 is a perspective view of one embodiment of a surgical instrumentincluding a plurality of fiducial markers;

FIG. 52 is a schematic view of one embodiment of intraoperative C-arm CTimaging;

FIG. 53 is a schematic view of one embodiment of a camera and a scopepositioned relative to a lung;

FIG. 54 is illustrates one embodiment of controlling energy of amicrowave ablation device;

FIG. 55 is a perspective view of a distal portion of one embodiment ofan ablation device in a compressed configuration;

FIG. 56 is a perspective view of the distal portion the ablation deviceof FIG. 55 in an expanded configuration;

FIG. 57 is a perspective view of the ablation device of FIG. 56positioned relative to a tumor;

FIG. 58 is a schematic partially cross-sectional view of one embodimentof a tissue lumen having a flexible scope positioned therein and alaparoscope positioned outside thereof;

FIG. 59 is a schematic cross-sectional view of one embodiment of aflexible force probe pressing on a tissue wall in which a scope ispositioned; and

FIG. 60 is a perspective view of the probe and the scope of FIG. 59 .

DETAILED DESCRIPTION

Certain exemplary embodiments will now be described to provide anoverall understanding of the principles of the structure, function,manufacture, and use of the devices, systems, and methods disclosedherein. One or more examples of these embodiments are illustrated in theaccompanying drawings. A person skilled in the art will understand thatthe devices, systems, and methods specifically described herein andillustrated in the accompanying drawings are non-limiting exemplaryembodiments and that the scope of the present invention is definedsolely by the claims. The features illustrated or described inconnection with one exemplary embodiment may be combined with thefeatures of other embodiments. Such modifications and variations areintended to be included within the scope of the present invention.

Further, in the present disclosure, like-named components of theembodiments generally have similar features, and thus within aparticular embodiment each feature of each like-named component is notnecessarily fully elaborated upon. Additionally, to the extent thatlinear or circular dimensions are used in the description of thedisclosed systems, devices, and methods, such dimensions are notintended to limit the types of shapes that can be used in conjunctionwith such systems, devices, and methods. A person skilled in the artwill recognize that an equivalent to such linear and circular dimensionscan easily be determined for any geometric shape. A person skilled inthe art will appreciate that a dimension or other measurement may not bea precise value but nevertheless be considered to be at about that valuedue to any number of factors such as manufacturing tolerances andsensitivity of measurement equipment. Sizes and shapes of the systemsand devices, and the components thereof, can depend at least on the sizeand shape of components with which the systems and devices will be used.

Surgical Visualization

In general, a surgical visualization system is configured to leverage“digital surgery” to obtain additional information about a patient’sanatomy and/or a surgical procedure. The surgical visualization systemis further configured to convey data to one or more medicalpractitioners in a helpful manner. Various aspects of the presentdisclosure provide improved visualization of the patient’s anatomyand/or the surgical procedure, and/or use visualization to provideimproved control of a surgical tool (also referred to herein as a“surgical device” or a “surgical instrument”).

“Digital surgery” can embrace robotic systems, advanced imaging,advanced instrumentation, artificial intelligence, machine learning,data analytics for performance tracking and benchmarking, connectivityboth inside and outside of the operating room (OR), and more. Althoughvarious surgical visualization systems described herein can be used incombination with a robotic surgical system, surgical visualizationsystems are not limited to use with a robotic surgical system. Incertain instances, surgical visualization using a surgical visualizationsystem can occur without robotics and/or with limited and/or optionalrobotic assistance. Similarly, digital surgery can occur withoutrobotics and/or with limited and/or optional robotic assistance.

In certain instances, a surgical system that incorporates a surgicalvisualization system may enable smart dissection in order to identifyand avoid critical structures. Critical structures include anatomicalstructures such as a ureter, an artery such as a superior mesentericartery, a vein such as a portal vein, a nerve such as a phrenic nerve,and/or a tumor, among other anatomical structures. In other instances, acritical structure can be a foreign structure in the anatomical field,such as a surgical device, a surgical fastener, a clip, a tack, abougie, a band, a plate, and other foreign structures. Criticalstructures can be determined on a patient-by-patient and/or aprocedure-by-procedure basis. Smart dissection technology may provide,for example, improved intraoperative guidance for dissection and/or mayenable smarter decisions with critical anatomy detection and avoidancetechnology.

A surgical system incorporating a surgical visualization system mayenable smart anastomosis technologies that provide more consistentanastomoses at optimal location(s) with improved workflow. Cancerlocalization technologies may be improved with a surgical visualizationplatform. For example, cancer localization technologies can identify andtrack a cancer location, orientation, and its margins. In certaininstances, the cancer localization technologies may compensate formovement of a surgical instrument, a patient, and/or the patient’sanatomy during a surgical procedure in order to provide guidance back tothe point of interest for medical practitioner(s).

A surgical visualization system may provide improved tissuecharacterization and/or lymph node diagnostics and mapping. For example,tissue characterization technologies may characterize tissue type andhealth without the need for physical haptics, especially when dissectingand/or placing stapling devices within the tissue. Certain tissuecharacterization technologies may be utilized without ionizing radiationand/or contrast agents. With respect to lymph node diagnostics andmapping, a surgical visualization platform may, for example,preoperatively locate, map, and ideally diagnose the lymph system and/orlymph nodes involved in cancerous diagnosis and staging.

During a surgical procedure, information available to a medicalpractitioner via the “naked eye” and/or an imaging system may provide anincomplete view of the surgical site. For example, certain structures,such as structures embedded or buried within an organ, can be at leastpartially concealed or hidden from view. Additionally, certaindimensions and/or relative distances can be difficult to ascertain withexisting sensor systems and/or difficult for the “naked eye” toperceive. Moreover, certain structures can move pre-operatively (e.g.,before a surgical procedure but after a preoperative scan) and/orintraoperatively. In such instances, the medical practitioner can beunable to accurately determine the location of a critical structureintraoperatively.

When the position of a critical structure is uncertain and/or when theproximity between the critical structure and a surgical tool is unknown,a medical practitioner’s decision-making process can be inhibited. Forexample, a medical practitioner may avoid certain areas in order toavoid inadvertent dissection of a critical structure; however, theavoided area may be unnecessarily large and/or at least partiallymisplaced. Due to uncertainty and/or overly/excessive exercises incaution, the medical practitioner may not access certain desiredregions. For example, excess caution may cause a medical practitioner toleave a portion of a tumor and/or other undesirable tissue in an effortto avoid a critical structure even if the critical structure is not inthe particular area and/or would not be negatively impacted by themedical practitioner working in that particular area. In certaininstances, surgical results can be improved with increased knowledgeand/or certainty, which can allow a surgeon to be more accurate and, incertain instances, less conservative/more aggressive with respect toparticular anatomical areas.

A surgical visualization system can allow for intraoperativeidentification and avoidance of critical structures. The surgicalvisualization system may thus enable enhanced intraoperative decisionmaking and improved surgical outcomes. The surgical visualization systemcan provide advanced visualization capabilities beyond what a medicalpractitioner sees with the “naked eye” and/or beyond what an imagingsystem can recognize and/or convey to the medical practitioner. Thesurgical visualization system can augment and enhance what a medicalpractitioner is able to know prior to tissue treatment (e.g.,dissection, etc.) and, thus, may improve outcomes in various instances.As a result, the medical practitioner can confidently maintain momentumthroughout the surgical procedure knowing that the surgicalvisualization system is tracking a critical structure, which may beapproached during dissection, for example. The surgical visualizationsystem can provide an indication to the medical practitioner insufficient time for the medical practitioner to pause and/or slow downthe surgical procedure and evaluate the proximity to the criticalstructure to prevent inadvertent damage thereto. The surgicalvisualization system can provide an ideal, optimized, and/orcustomizable amount of information to the medical practitioner to allowthe medical practitioner to move confidently and/or quickly throughtissue while avoiding inadvertent damage to healthy tissue and/orcritical structure(s) and, thus, to minimize the risk of harm resultingfrom the surgical procedure.

Surgical visualization systems are described in detail below. Ingeneral, a surgical visualization system can include a first lightemitter configured to emit a plurality of spectral waves, a second lightemitter configured to emit a light pattern, and a receiver, or sensor,configured to detect visible light, molecular responses to the spectralwaves (spectral imaging), and/or the light pattern. The surgicalvisualization system can also include an imaging system and a controlcircuit in signal communication with the receiver and the imagingsystem. Based on output from the receiver, the control circuit candetermine a geometric surface map, e.g., three-dimensional surfacetopography, of the visible surfaces at the surgical site and a distancewith respect to the surgical site, such as a distance to an at leastpartially concealed structure. The imaging system can convey thegeometric surface map and the distance to a medical practitioner. Insuch instances, an augmented view of the surgical site provided to themedical practitioner can provide a representation of the concealedstructure within the relevant context of the surgical site. For example,the imaging system can virtually augment the concealed structure on thegeometric surface map of the concealing and/or obstructing tissuesimilar to a line drawn on the ground to indicate a utility line belowthe surface. Additionally or alternatively, the imaging system canconvey the proximity of a surgical tool to the visible and obstructingtissue and/or to the at least partially concealed structure and/or adepth of the concealed structure below the visible surface of theobstructing tissue. For example, the visualization system can determinea distance with respect to the augmented line on the surface of thevisible tissue and convey the distance to the imaging system.

Throughout the present disclosure, any reference to “light,” unlessspecifically in reference to visible light, can include electromagneticradiation (EMR) or photons in the visible and/or non-visible portions ofthe EMR wavelength spectrum. The visible spectrum, sometimes referred toas the optical spectrum or luminous spectrum, is that portion of theelectromagnetic spectrum that is visible to (e.g., can be detected by)the human eye and may be referred to as “visible light” or simply“light.” A typical human eye will respond to wavelengths in air that arefrom about 380 nm to about 750 nm. The invisible spectrum (e.g., thenon-luminous spectrum) is that portion of the electromagnetic spectrumthat lies below and above the visible spectrum. The invisible spectrumis not detectable by the human eye. Wavelengths greater than about 750nm are longer than the red visible spectrum, and they become invisibleinfrared (IR), microwave, and radio electromagnetic radiation.Wavelengths less than about 380 nm are shorter than the violet spectrum,and they become invisible ultraviolet, x-ray, and gamma rayelectromagnetic radiation.

FIG. 1 illustrates one embodiment of a surgical visualization system100. The surgical visualization system 100 is configured to create avisual representation of a critical structure 101 within an anatomicalfield. The critical structure 101 can include a single criticalstructure or a plurality of critical structures. As discussed herein,the critical structure 101 can be any of a variety of structures, suchas an anatomical structure, e.g., a ureter, an artery such as a superiormesenteric artery, a vein such as a portal vein, a nerve such as aphrenic nerve, a vessel, a tumor, or other anatomical structure, or aforeign structure, e.g., a surgical device, a surgical fastener, asurgical clip, a surgical tack, a bougie, a surgical band, a surgicalplate, or other foreign structure. As discussed herein, the criticalstructure 101 can be identified on a patient-by-patient and/or aprocedure-by-procedure basis. Embodiments of critical structures and ofidentifying critical structures using a visualization system are furtherdescribed in U.S. Pat. No. 10,792,034 entitled “Visualization OfSurgical Devices” issued Oct. 6, 2020, which is hereby incorporated byreference in its entirety.

In some instances, the critical structure 101 can be embedded in tissue103. The tissue 103 can be any of a variety of tissues, such as fat,connective tissue, adhesions, and/or organs. Stated differently, thecritical structure 101 may be positioned below a surface 105 of thetissue 103. In such instances, the tissue 103 conceals the criticalstructure 101 from the medical practitioner’s “naked eye” view. Thetissue 103 also obscures the critical structure 101 from the view of animaging device 120 of the surgical visualization system 100. Instead ofbeing fully obscured, the critical structure 101 can be partiallyobscured from the view of the medical practitioner and/or the imagingdevice 120.

The surgical visualization system 100 can be used for clinical analysisand/or medical intervention. In certain instances, the surgicalvisualization system 100 can be used intraoperatively to providereal-time information to the medical practitioner during a surgicalprocedure, such as real-time information regarding proximity data,dimensions, and/or distances. A person skilled in the art willappreciate that information may not be precisely real time butnevertheless be considered to be real time for any of a variety ofreasons, such as time delay induced by data transmission, time delayinduced by data processing, and/or sensitivity of measurement equipment.The surgical visualization system 100 is configured for intraoperativeidentification of critical structure(s) and/or to facilitate theavoidance of the critical structure(s) 101 by a surgical device. Forexample, by identifying the critical structure 101, a medicalpractitioner can avoid maneuvering a surgical device around the criticalstructure 101 and/or a region in a predefined proximity of the criticalstructure 101 during a surgical procedure. For another example, byidentifying the critical structure 101, a medical practitioner can avoiddissection of and/or near the critical structure 101, thereby helping toprevent damage to the critical structure 101 and/or helping to prevent asurgical device being used by the medical practitioner from beingdamaged by the critical structure 101.

The surgical visualization system 100 is configured to incorporatetissue identification and geometric surface mapping in combination withthe surgical visualization system’s distance sensor system 104. Incombination, these features of the surgical visualization system 100 candetermine a position of a critical structure 101 within the anatomicalfield and/or the proximity of a surgical device 102 to the surface 105of visible tissue 103 and/or to the critical structure 101. Moreover,the surgical visualization system 100 includes an imaging system thatincludes the imaging device 120 configured to provide real-time views ofthe surgical site. The imaging device 120 can include, for example, aspectral camera (e.g., a hyperspectral camera, multispectral camera, orselective spectral camera), which is configured to detect reflectedspectral waveforms and generate a spectral cube of images based on themolecular response to the different wavelengths. Views from the imagingdevice 120 can be provided in real time to a medical practitioner, suchas on a display (e.g., a monitor, a computer tablet screen, etc.). Thedisplayed views can be augmented with additional information based onthe tissue identification, landscape mapping, and the distance sensorsystem 104. In such instances, the surgical visualization system 100includes a plurality of subsystems—an imaging subsystem, a surfacemapping subsystem, a tissue identification subsystem, and/or a distancedetermining subsystem. These subsystems can cooperate tointra-operatively provide advanced data synthesis and integratedinformation to the medical practitioner.

The imaging device 120 can be configured to detect visible light,spectral light waves (visible or invisible), and a structured lightpattern (visible or invisible). Examples of the imaging device 120includes scopes, e.g., an endoscope, an arthroscope, an angioscope, abronchoscope, a choledochoscope, a colonoscope, a cytoscope, aduodenoscope, an enteroscope, an esophagogastroduodenoscope(gastroscope), a laryngoscope, a nasopharyngo-neproscope, asigmoidoscope, a thoracoscope, an ureteroscope, or an exoscope. Scopescan be particularly useful in minimally invasive surgical procedures. Inopen surgery applications, the imaging device 120 may not include ascope.

The tissue identification subsystem can be achieved with a spectralimaging system. The spectral imaging system can rely on imaging such ashyperspectral imaging, multispectral imaging, or selective spectralimaging. Embodiments of hyperspectral imaging of tissue are furtherdescribed in U.S. Pat. No. 9,274,047 entitled “System And Method ForGross Anatomic Pathology Using Hyperspectral Imaging” issued Mar. 1,2016, which is hereby incorporated by reference in its entirety.

The surface mapping subsystem can be achieved with a light patternsystem. Various surface mapping techniques using a light pattern (orstructured light) for surface mapping can be utilized in the surgicalvisualization systems described herein. Structured light is the processof projecting a known pattern (often a grid or horizontal bars) on to asurface. In certain instances, invisible (or imperceptible) structuredlight can be utilized, in which the structured light is used withoutinterfering with other computer vision tasks for which the projectedpattern may be confusing. For example, infrared light or extremely fastframe rates of visible light that alternate between two exact oppositepatterns can be utilized to prevent interference. Embodiments of surfacemapping and a surgical system including a light source and a projectorfor projecting a light pattern are further described in U.S. Pat. Pub.No. 2017/0055819 entitled “Set Comprising A Surgical Instrument”published Mar. 2, 2017, U.S. Pat. Pub. No. 2017/0251900 entitled“Depiction System” published Sep. 7, 2017, and U.S. Pat. Pub. No.2021/0196385 entitled “Surgical Systems For Generating Three DimensionalConstructs Of Anatomical Organs And Coupling Identified AnatomicalStructures Thereto” published Jul. 1, 2021, which are herebyincorporated by reference in their entireties.

The distance determining system can be incorporated into the surfacemapping system. For example, structured light can be utilized togenerate a three-dimensional (3D) virtual model of the visible surface105 and determine various distances with respect to the visible surface105. Additionally or alternatively, the distance determining system canrely on time-of-flight measurements to determine one or more distancesto the identified tissue (or other structures) at the surgical site.

The surgical visualization system 100 also includes a surgical device102. The surgical device 102 can be any suitable surgical device.Examples of the surgical device 102 includes a surgical dissector, asurgical stapler, a surgical grasper, a clip applier, a smoke evacuator,a surgical energy device (e.g., mono-polar probes, bi-polar probes,ablation probes, an ultrasound device, an ultrasonic end effector,etc.), etc. In some embodiments, the surgical device 102 includes an endeffector having opposing jaws that extend from a distal end of a shaftof the surgical device 102 and that are configured to engage tissuetherebetween.

The surgical visualization system 100 can be configured to identify thecritical structure 101 and a proximity of the surgical device 102 to thecritical structure 101. The imaging device 120 of the surgicalvisualization system 100 is configured to detect light at variouswavelengths, such as visible light, spectral light waves (visible orinvisible), and a structured light pattern (visible or invisible). Theimaging device 120 can include a plurality of lenses, sensors, and/orreceivers for detecting the different signals. For example, the imagingdevice 120 can be a hyperspectral, multispectral, or selective spectralcamera, as described herein. The imaging device 120 can include awaveform sensor 122 (such as a spectral image sensor, detector, and/orthree-dimensional camera lens). For example, the imaging device 120 caninclude a right-side lens and a left-side lens used together to recordtwo two-dimensional images at the same time and, thus, generate a 3Dimage of the surgical site, render a three-dimensional image of thesurgical site, and/or determine one or more distances at the surgicalsite. Additionally or alternatively, the imaging device 120 can beconfigured to receive images indicative of the topography of the visibletissue and the identification and position of hidden criticalstructures, as further described herein. For example, a field of view ofthe imaging device 120 can overlap with a pattern of light (structuredlight) on the surface 105 of the tissue 103, as shown in FIG. 1 .

As in this illustrated embodiment, the surgical visualization system 100can be incorporated into a robotic surgical system 110. The roboticsurgical system 110 can have a variety of configurations, as discussedherein. In this illustrated embodiment, the robotic surgical system 110includes a first robotic arm 112 and a second robotic arm 114. Therobotic arms 112, 114 each include rigid structural members 116 andjoints 118, which can include servomotor controls. The first robotic arm112 is configured to maneuver the surgical device 102, and the secondrobotic arm 114 is configured to maneuver the imaging device 120. Arobotic control unit of the robotic surgical system 110 is configured toissue control motions to the first and second robotic arms 112, 114,which can affect the surgical device 102 and the imaging device 120,respectively.

In some embodiments, one or more of the robotic arms 112, 114 can beseparate from the main robotic system 110 used in the surgicalprocedure. For example, at least one of the robotic arms 112, 114 can bepositioned and registered to a particular coordinate system without aservomotor control. For example, a closed-loop control system and/or aplurality of sensors for the robotic arms 112, 114 can control and/orregister the position of the robotic arm(s) 112, 114 relative to theparticular coordinate system. Similarly, the position of the surgicaldevice 102 and the imaging device 120 can be registered relative to aparticular coordinate system.

Examples of robotic surgical systems include the Ottava™robotic-assisted surgery system (Johnson & Johnson of New Brunswick,NJ), da Vinci^(®) surgical systems (Intuitive Surgical, Inc. ofSunnyvale, CA), the Hugo™ robotic-assisted surgery system (Medtronic PLCof Minneapolis, MN), the Versius^(®) surgical robotic system (CMRSurgical Ltd of Cambridge, UK), and the Monarch^(®) platform (AurisHealth, Inc. of Redwood City, CA). Embodiments of various roboticsurgical systems and using robotic surgical systems are furtherdescribed in U.S. Pat. Pub. No. 2018/0177556 entitled “FlexibleInstrument Insertion Using An Adaptive Force Threshold” filed Dec. 28,2016, U.S. Pat. Pub. No. 2020/0000530 entitled “Systems And TechniquesFor Providing Multiple Perspectives During Medical Procedures” filedApr. 16, 2019, U.S. Pat. Pub. No. 2020/0170720 entitled “Image-BasedBranch Detection And Mapping For Navigation” filed Feb. 7, 2020, U.S.Pat. Pub. No. 2020/0188043 entitled “Surgical Robotics System” filedDec. 9, 2019, U.S. Pat. Pub. No. 2020/0085516 entitled “Systems AndMethods For Concomitant Medical Procedures” filed Sep. 3, 2019, U.S.Pat. No. 8,831,782 entitled “Patient-Side Surgeon Interface For ATeleoperated Surgical Instrument” filed Jul. 15, 2013, and Intl. Pat.Pub. No. WO 2014151621 entitled “Hyperdexterous Surgical System” filedMar. 13, 2014, which are hereby incorporated by reference in theirentireties.

The surgical visualization system 100 also includes an emitter 106. Theemitter 106 is configured to emit a pattern of light, such as stripes,grid lines, and/or dots, to enable the determination of the topographyor landscape of the surface 105. For example, projected light arrays 130can be used for three-dimensional scanning and registration on thesurface 105. The projected light arrays 130 can be emitted from theemitter 106 located on the surgical device 102 and/or one of the roboticarms 112, 114 and/or the imaging device 120. In one aspect, theprojected light array 130 is employed by the surgical visualizationsystem 100 to determine the shape defined by the surface 105 of thetissue 103 and/or motion of the surface 105 intraoperatively. Theimaging device 120 is configured to detect the projected light arrays130 reflected from the surface 105 to determine the topography of thesurface 105 and various distances with respect to the surface 105.

As in this illustrated embodiment, the imaging device 120 can include anoptical waveform emitter 123, such as by being mounted on or otherwiseattached on the imaging device 120. The optical waveform emitter 123 isconfigured to emit electromagnetic radiation 124 (near-infrared (NIR)photons) that can penetrate the surface 105 of the tissue 103 and reachthe critical structure 101. The imaging device 120 and the opticalwaveform emitter 123 can be positionable by the robotic arm 114. Theoptical waveform emitter 123 is mounted on or otherwise on the imagingdevice 120 but in other embodiments can be positioned on a separatesurgical device from the imaging device 120. A corresponding waveformsensor 122 (e.g., an image sensor, spectrometer, or vibrational sensor)of the imaging device 120 is configured to detect the effect of theelectromagnetic radiation received by the waveform sensor 122. Thewavelengths of the electromagnetic radiation 124 emitted by the opticalwaveform emitter 123 are configured to enable the identification of thetype of anatomical and/or physical structure, such as the criticalstructure 101. The identification of the critical structure 101 can beaccomplished through spectral analysis, photo-acoustics, and/orultrasound, for example. In one aspect, the wavelengths of theelectromagnetic radiation 124 can be variable. The waveform sensor 122and optical waveform emitter 123 can be inclusive of a multispectralimaging system and/or a selective spectral imaging system, for example.In other instances, the waveform sensor 122 and optical waveform emitter123 can be inclusive of a photoacoustic imaging system, for example.

The distance sensor system 104 of the surgical visualization system 100is configured to determine one or more distances at the surgical site.The distance sensor system 104 can be a time-of-flight distance sensorsystem that includes an emitter, such as the emitter 106 as in thisillustrated embodiment, and that includes a receiver 108. In otherinstances, the time-of-flight emitter can be separate from thestructured light emitter. The emitter 106 can include a very tiny lasersource, and the receiver 108 can include a matching sensor. The distancesensor system 104 is configured to detect the “time of flight,” or howlong the laser light emitted by the emitter 106 has taken to bounce backto the sensor portion of the receiver 108. Use of a very narrow lightsource in the emitter 106 enables the distance sensor system 104 todetermining the distance to the surface 105 of the tissue 103 directlyin front of the distance sensor system 104.

The receiver 108 of the distance sensor system 104 is positioned on thesurgical device 102 in this illustrated embodiment, but in otherembodiments the receiver 108 can be mounted on a separate surgicaldevice instead of the surgical device 102. For example, the receiver 108can be mounted on a cannula or trocar through which the surgical device102 extends to reach the surgical site. In still other embodiments, thereceiver 108 for the distance sensor system 104 can be mounted on aseparate robotically-controlled arm of the robotic system 110 (e.g., onthe second robotic arm 114) than the first robotic arm 112 to which thesurgical device 102 is coupled, can be mounted on a movable arm that isoperated by another robot, or be mounted to an operating room (OR) tableor fixture. In some embodiments, the imaging device 120 includes thereceiver 108 to allow for determining the distance from the emitter 106to the surface 105 of the tissue 103 using a line between the emitter106 on the surgical device 102 and the imaging device 120. For example,a distance d_(e) can be triangulated based on known positions of theemitter 106 (on the surgical device 102) and the receiver 108 (on theimaging device 120) of the distance sensor system 104. The 3D positionof the receiver 108 can be known and/or registered to the robotcoordinate plane intraoperatively.

As in this illustrated embodiment, the position of the emitter 106 ofthe distance sensor system 104 can be controlled by the first roboticarm 112, and the position of the receiver 108 of the distance sensorsystem 104 can be controlled by the second robotic arm 114. In otherembodiments, the surgical visualization system 100 can be utilized apartfrom a robotic system. In such instances, the distance sensor system 104can be independent of the robotic system.

In FIG. 1 , distance d_(e) is emitter-to-tissue distance from theemitter 106 to the surface 105 of the tissue 103, and distance d_(t) isdevice-to-tissue distance from a distal end of the surgical device 102to the surface 105 of the tissue 103. The distance sensor system 104 isconfigured to determine the emitter-to-tissue distance d_(e). Thedevice-to-tissue distance d_(t) is obtainable from the known position ofthe emitter 106 on the surgical device 102, e.g., on a shaft thereofproximal to the surgical device’s distal end, relative to the distal endof the surgical device 102. In other words, when the distance betweenthe emitter 106 and the distal end of the surgical device 102 is known,the device-to-tissue distance d_(t) can be determined from theemitter-to-tissue distance d_(e). In some embodiments, the shaft of thesurgical device 102 can include one or more articulation joints and canbe articulatable with respect to the emitter 106 and jaws at the distalend of the surgical device 102. The articulation configuration caninclude a multi-joint vertebrae-like structure, for example. In someembodiments, a 3D camera can be utilized to triangulate one or moredistances to the surface 105.

In FIG. 1 , distance d_(w) is camera-to-critical structure distance fromthe optical waveform emitter 123 located on the imaging device 120 tothe surface of the critical structure 101, and distance d_(A) is a depthof the critical structure 101 below the surface 105 of the tissue 103(e.g., the distance between the portion of the surface 105 closest tothe surgical device 102 and the critical structure 101). Thetime-of-flight of the optical waveforms emitted from the opticalwaveform emitter 123 located on the imaging device 120 are configured todetermine the camera-to-critical structure distance d_(w).

As shown in FIG. 2 , the depth d_(A) of the critical structure 101relative to the surface 105 of the tissue 103 can be determined bytriangulating from the camera-to-critical structure distance d_(w) andknown positions of the emitter 106 on the surgical device 102 and theoptical waveform emitter 123 on the imaging device 120 (and, thus, theknown distance d_(x) therebetween) to determine distance d_(y), which isthe sum of the distances d_(e) and d_(A). Additionally or alternatively,time-of-flight from the optical waveform emitter 123 can be configuredto determine the distance from the optical waveform emitter 123 to thesurface 105 of the tissue 103. For example, a first waveform (or rangeof waveforms) can be utilized to determine the camera-to-criticalstructure distance d_(w) and a second waveform (or range of waveforms)can be utilized to determine the distance to the surface 105 of thetissue 103. In such instances, the different waveforms can be utilizedto determine the depth of the critical structure 101 below the surface105 of the tissue 103.

Additionally or alternatively, the distance d_(A) can be determined froman ultrasound, a registered magnetic resonance imaging (MRI), orcomputerized tomography (CT) scan. In still other instances, thedistance d_(A) can be determined with spectral imaging because thedetection signal received by the imaging device 120 can vary based onthe type of material, e.g., type of the tissue 103. For example, fat candecrease the detection signal in a first way, or a first amount, andcollagen can decrease the detection signal in a different, second way,or a second amount.

In another embodiment of a surgical visualization system 160 illustratedin FIG. 3 , a surgical device 162, and not the imaging device 120,includes the optical waveform emitter 123 and the waveform sensor 122that is configured to detect the reflected waveforms. The opticalwaveform emitter 123 is configured to emit waveforms for determining thedistances d_(t) and d_(w) from a common device, such as the surgicaldevice 162, as described herein. In such instances, the distance d_(A)from the surface 105 of the tissue 103 to the surface of the criticalstructure 101 can be determined as follows:

d_(A) = d_(w)- d_(t)

The surgical visualization system 100 includes a control systemconfigured to control various aspects of the surgical visualizationsystem 100. FIG. 4 illustrates one embodiment of a control system 133that can be utilized as the control system of the surgical visualizationsystem 100 (or other surgical visualization system described herein).The control system 133 includes a control circuit 132 configured to bein signal communication with a memory 134. The memory 134 is configuredto store instructions executable by the control circuit 132, such asinstructions to determine and/or recognize critical structures (e.g.,the critical structure 101 of FIG. 1 ), instructions to determine and/orcompute one or more distances and/or three-dimensional digitalrepresentations, and instructions to communicate information to amedical practitioner. As in this illustrated embodiment, the memory 134can store surface mapping logic 136, imaging logic 138, tissueidentification logic 140, and distance determining logic 141, althoughthe memory 134 can store any combinations of the logics 136, 138, 140,141 and/or can combine various logics together. The control system 133also includes an imaging system 142 including a camera 144 (e.g., theimaging system including the imaging device 120 of FIG. 1 ), a display146 (e.g., a monitor, a computer tablet screen, etc.), and controls 148of the camera 144 and the display 146. The camera 144 includes an imagesensor 135 (e.g., the waveform sensor 122) configured to receive signalsfrom various light sources emitting light at various visible andinvisible spectra (e.g., visible light, spectral imagers,three-dimensional lens, etc.). The display 146 is configured to depictreal, virtual, and/or virtually-augmented images and/or information to amedical practitioner.

In an exemplary embodiment, the image sensor 135 is a solid-stateelectronic device containing up to millions of discrete photodetectorsites called pixels. The image sensor 135 technology falls into one oftwo categories: Charge-Coupled Device (CCD) and Complementary MetalOxide Semiconductor (CMOS) imagers and more recently, short-waveinfrared (SWIR) is an emerging technology in imaging. Another type ofthe image sensor 135 employs a hybrid CCD/CMOS architecture (sold underthe name “sCMOS”) and consists of CMOS readout integrated circuits(ROICs) that are bump bonded to a CCD imaging substrate. CCD and CMOSimage sensors are sensitive to wavelengths in a range of about 350 nm toabout 1050 nm, such as in a range of about 400 nm to about 1000 nm. Aperson skilled in the art will appreciate that a value may not beprecisely at a value but nevertheless considered to be about that valuefor any of a variety of reasons, such as sensitivity of measurementequipment and manufacturing tolerances. CMOS sensors are, in general,more sensitive to IR wavelengths than CCD sensors. Solid state imagesensors are based on the photoelectric effect and, as a result, cannotdistinguish between colors. Accordingly, there are two types of colorCCD cameras: single chip and three-chip. Single chip color CCD camerasoffer a common, low-cost imaging solution and use a mosaic (e.g., Bayer)optical filter to separate incoming light into a series of colors andemploy an interpolation algorithm to resolve full color images. Eachcolor is, then, directed to a different set of pixels. Three-chip colorCCD cameras provide higher resolution by employing a prism to directeach section of the incident spectrum to a different chip. More accuratecolor reproduction is possible, as each point in space of the object hasseparate RGB intensity values, rather than using an algorithm todetermine the color. Three-chip cameras offer extremely highresolutions.

The control system 133 also includes an emitter (e.g., the emitter 106)including a spectral light source 150 and a structured light source 152each operably coupled to the control circuit 133. A single source can bepulsed to emit wavelengths of light in the spectral light source 150range and wavelengths of light in the structured light source 152 range.Alternatively, a single light source can be pulsed to provide light inthe invisible spectrum (e.g., infrared spectral light) and wavelengthsof light on the visible spectrum. The spectral light source 150 can be,for example, a hyperspectral light source, a multispectral light source,and/or a selective spectral light source. The tissue identificationlogic 140 is configured to identify critical structure(s) (e.g., thecritical structure 101 of FIG. 1 ) via data from the spectral lightsource 150 received by the image sensor 135 of the camera 144. Thesurface mapping logic 136 is configured to determine the surfacecontours of the visible tissue (e.g., the tissue 103) based on reflectedstructured light. With time-of-flight measurements, the distancedetermining logic 141 is configured to determine one or more distance(s)to the visible tissue and/or the critical structure. Output from each ofthe surface mapping logic 136, the tissue identification logic 140, andthe distance determining logic 141 is configured to be provided to theimaging logic 138, and combined, blended, and/or overlaid by the imaginglogic 138 to be conveyed to a medical practitioner via the display 146of the imaging system 142.

The control circuit 132 can have a variety of configurations. FIG. 5illustrates one embodiment of a control circuit 170 that can be used asthe control circuit 132 configured to control aspects of the surgicalvisualization system 100. The control circuit 170 is configured toimplement various processes described herein. The control circuit 170includes a microcontroller that includes a processor 172 (e.g., amicroprocessor or microcontroller) operably coupled to a memory 174. Thememory 174 is configured to store machine-executable instructions that,when executed by the processor 172, cause the processor 172 to executemachine instructions to implement various processes described herein.The processor 172 can be any one of a number of single-core or multicoreprocessors known in the art. The memory 174 can include volatile andnon-volatile storage media. The processor 172 includes an instructionprocessing unit 176 and an arithmetic unit 178. The instructionprocessing unit 176 is configured to receive instructions from thememory 174.

The surface mapping logic 136, the imaging logic 138, the tissueidentification logic 140, and the distance determining logic 141 canhave a variety of configurations. FIG. 6 illustrates one embodiment of acombinational logic circuit 180 configured to control aspects of thesurgical visualization system 100 using logic such as one or more of thesurface mapping logic 136, the imaging logic 138, the tissueidentification logic 140, and the distance determining logic 141. Thecombinational logic circuit 180 includes a finite state machine thatincludes a combinational logic 182 configured to receive data associatedwith a surgical device (e.g. the surgical device 102 and/or the imagingdevice 120) at an input 184, process the data by the combinational logic182, and provide an output 184 to a control circuit (e.g., the controlcircuit 132).

FIG. 7 illustrates one embodiment of a sequential logic circuit 190configured to control aspects of the surgical visualization system 100using logic such as one or more of the surface mapping logic 136, theimaging logic 138, the tissue identification logic 140, and the distancedetermining logic 141. The sequential logic circuit 190 includes afinite state machine that includes a combinational logic 192, a memory194, and a clock 196. The memory 194 is configured to store a currentstate of the finite state machine. The sequential logic circuit 190 canbe synchronous or asynchronous. The combinational logic 192 isconfigured to receive data associated with a surgical device (e.g. thesurgical device 102 and/or the imaging device 120) at an input 426,process the data by the combinational logic 192, and provide an output499 to a control circuit (e.g., the control circuit 132). In someembodiments, the sequential logic circuit 190 can include a combinationof a processor (e.g., processor 172 of FIG. 5 ) and a finite statemachine to implement various processes herein. In some embodiments, thefinite state machine can include a combination of a combinational logiccircuit (e.g., the combinational logic circuit 192 of FIG. 7 ) and thesequential logic circuit 190.

FIG. 8 illustrates another embodiment of a surgical visualization system200. The surgical visualization system 200 is generally configured andused similar to the surgical visualization system 100 of FIG. 1 , e.g.,includes a surgical device 202 and an imaging device 220. The imagingdevice 220 includes a spectral light emitter 223 configured to emitspectral light in a plurality of wavelengths to obtain a spectral imageof hidden structures, for example. The imaging device 220 can alsoinclude a three-dimensional camera and associated electronic processingcircuits. The surgical visualization system 200 is shown being utilizedintraoperatively to identify and facilitate avoidance of certaincritical structures, such as a ureter 201 a and vessels 201 b, in anorgan 203 (a uterus in this embodiment) that are not visible on asurface 205 of the organ 203.

The surgical visualization system 200 is configured to determine anemitter-to-tissue distance d_(e) from an emitter 206 on the surgicaldevice 202 to the surface 205 of the uterus 203 via structured light.The surgical visualization system 200 is configured to extrapolate adevice-to-tissue distance d_(t) from the surgical device 202 to thesurface 205 of the uterus 203 based on the emitter-to-tissue distanced_(e). The surgical visualization system 200 is also configured todetermine a tissue-to-ureter distance d_(A) from the ureter 201 a to thesurface 205 and a camera-to ureter distance d_(w) from the imagingdevice 220 to the ureter 201 a. As described herein, e.g., with respectto the surgical visualization system 100 of FIG. 1 , the surgicalvisualization system 200 is configured to determine the distance d_(w)with spectral imaging and time-of-flight sensors, for example. Invarious embodiments, the surgical visualization system 200 can determine(e.g., triangulate) the tissue-to-ureter distance d_(A) (or depth) basedon other distances and/or the surface mapping logic described herein.

As mentioned above, a surgical visualization system includes a controlsystem configured to control various aspects of the surgicalvisualization system. The control system can have a variety ofconfigurations. FIG. 9 illustrates one embodiment of a control system600 for a surgical visualization system, such as the surgicalvisualization system 100 of FIG. 1 , the surgical visualization system200 of FIG. 8 , or other surgical visualization system described herein.The control system 600 is a conversion system that integrates spectralsignature tissue identification and structured light tissue positioningto identify a critical structure, especially when those structure(s) areobscured by tissue, e.g., by fat, connective tissue, blood tissue,and/or organ(s), and/or by blood, and/or to detect tissue variability,such as differentiating tumors and/or non-healthy tissue from healthytissue within an organ.

The control system 600 is configured for implementing a hyperspectralimaging and visualization system in which a molecular response isutilized to detect and identify anatomy in a surgical field of view. Thecontrol system 600 includes a conversion logic circuit 648 configured toconvert tissue data to usable information for surgeons and/or othermedical practitioners. For example, variable reflectance based onwavelengths with respect to obscuring material can be utilized toidentify the critical structure in the anatomy. Moreover, the controlsystem 600 is configured to combine the identified spectral signatureand the structural light data in an image. For example, the controlsystem 600 can be employed to create of three-dimensional data set forsurgical use in a system with augmentation image overlays. Techniquescan be employed both intraoperatively and preoperatively usingadditional visual information. In various embodiments, the controlsystem 600 is configured to provide warnings to a medical practitionerwhen in the proximity of one or more critical structures. Variousalgorithms can be employed to guide robotic automation andsemi-automated approaches based on the surgical procedure and proximityto the critical structure(s).

A projected array of lights is employed by the control system 600 todetermine tissue shape and motion intraoperatively. Alternatively, flashLidar may be utilized for surface mapping of the tissue.

The control system 600 is configured to detect the critical structure,which as mentioned above can include one or more critical structures,and provide an image overlay of the critical structure and measure thedistance to the surface of the visible tissue and the distance to theembedded/buried critical structure(s). The control system 600 canmeasure the distance to the surface of the visible tissue or detect thecritical structure and provide an image overlay of the criticalstructure.

The control system 600 includes a spectral control circuit 602. Thespectral control circuit 602 can be a field programmable gate array(FPGA) or another suitable circuit configuration, such as theconfigurations described with respect to FIG. 6 , FIG. 7 , and FIG. 8 .The spectral control circuit 602 includes a processor 604 configured toreceive video input signals from a video input processor 606. Theprocessor 604 can be configured for hyperspectral processing and canutilize C/C++ code, for example. The video input processor 606 isconfigured to receive video-in of control (metadata) data such asshutter time, wave length, and sensor analytics, for example. Theprocessor 604 is configured to process the video input signal from thevideo input processor 606 and provide a video output signal to a videooutput processor 608, which includes a hyperspectral video-out ofinterface control (metadata) data, for example. The video outputprocessor 608 is configured to provides the video output signal to animage overlay controller 610.

The video input processor 606 is operatively coupled to a camera 612 atthe patient side via a patient isolation circuit 614. The camera 612includes a solid state image sensor 634. The patient isolation circuit614 can include a plurality of transformers so that the patient isisolated from other circuits in the system. The camera 612 is configuredto receive intraoperative images through optics 632 and the image sensor634. The image sensor 634 can include a CMOS image sensor, for example,or can include another image sensor technology, such as those discussedherein in connection with FIG. 4 . The camera 612 is configured tooutput 613 images in 14 bit / pixel signals. A person skilled in the artwill appreciate that higher or lower pixel resolutions can be employed.The isolated camera output signal 613 is provided to a color RGB fusioncircuit 616, which in this illustrated embodiment employs a hardwareregister 618 and a Nios2 co-processor 620 configured to process thecamera output signal 613. A color RGB fusion output signal is providedto the video input processor 606 and a laser pulsing control circuit622.

The laser pulsing control circuit 622 is configured to control a laserlight engine 624. The laser light engine 624 is configured to outputlight in a plurality of wavelengths (λ1, λ2, λ3, ... λn) including nearinfrared (NIR). The laser light engine 624 can operate in a plurality ofmodes. For example, the laser light engine 624 can operate in two modes.In a first mode, e.g., a normal operating mode, the laser light engine624 is configured to output an illuminating signal. In a second mode,e.g., an identification mode, the laser light engine 624 is configuredto output RGBG and NIR light. In various embodiments, the laser lightengine 624 can operate in a polarizing mode.

Light output 626 from the laser light engine 624 is configured toilluminate targeted anatomy in an intraoperative surgical site 627. Thelaser pulsing control circuit 622 is also configured to control a laserpulse controller 628 for a laser pattern projector 630 configured toproject a laser light pattern 631, such as a grid or pattern of linesand/or dots, at a predetermined wavelength (λ2) on an operative tissueor organ at the surgical site 627. The camera 612 is configured toreceive the patterned light as well as the reflected light outputthrough the camera optics 632. The image sensor 634 is configured toconvert the received light into a digital signal.

The color RGB fusion circuit 616 is also configured to output signals tothe image overlay controller 610 and a video input module 636 forreading the laser light pattern 631 projected onto the targeted anatomyat the surgical site 627 by the laser pattern projector 630. Aprocessing module 638 is configured to process the laser light pattern631 and output a first video output signal 640 representative of thedistance to the visible tissue at the surgical site 627. The data isprovided to the image overlay controller 610. The processing module 638is also configured to output a second video signal 642 representative ofa three-dimensional rendered shape of the tissue or organ of thetargeted anatomy at the surgical site.

The first and second video output signals 640, 642 include datarepresentative of the position of the critical structure on athree-dimensional surface model, which is provided to an integrationmodule 643. In combination with data from the video out processor 608 ofthe spectral control circuit 602, the integration module 643 isconfigured to determine the distance (e.g., distance d_(A) of FIG. 1 )to a buried critical structure (e.g., via triangularization algorithms644), and the distance to the buried critical structure can be providedto the image overlay controller 610 via a video out processor 646. Theforegoing conversion logic can encompass the conversion logic circuit648 intermediate video monitors 652 and the camera 624 / laser patternprojector 630 positioned at the surgical site 627.

Preoperative data 650, such as from a CT or MRI scan, can be employed toregister or align certain three-dimensional deformable tissue in variousinstances. Such preoperative data 650 can be provided to the integrationmodule 643 and ultimately to the image overlay controller 610 so thatsuch information can be overlaid with the views from the camera 612 andprovided to the video monitors 652. Embodiments of registration ofpreoperative data are further described in U.S. Pat. Pub. No.2020/0015907 entitled “Integration Of Imaging Data” filed Sep. 11, 2018,which is hereby incorporated by reference herein in its entirety.

The video monitors 652 are configured to output the integrated/augmentedviews from the image overlay controller 610. A medical practitioner canselect and/or toggle between different views on one or more displays. Ona first display 652 a, which is a monitor in this illustratedembodiment, the medical practitioner can toggle between (A) a view inwhich a three-dimensional rendering of the visible tissue is depictedand (B) an augmented view in which one or more hidden criticalstructures are depicted over the three-dimensional rendering of thevisible tissue. On a second display 652 b, which is a monitor in thisillustrated embodiment, the medical practitioner can toggle on distancemeasurements to one or more hidden critical structures and/or thesurface of visible tissue, for example.

The various surgical visualization systems described herein can beutilized to visualize various different types of tissues and/oranatomical structures, including tissues and/or anatomical structuresthat may be obscured from being visualized by EMR in the visible portionof the spectrum. The surgical visualization system can utilize aspectral imaging system, as mentioned above, which can be configured tovisualize different types of tissues based upon their varyingcombinations of constituent materials. In particular, a spectral imagingsystem can be configured to detect the presence of various constituentmaterials within a tissue being visualized based on the absorptioncoefficient of the tissue across various EMR wavelengths. The spectralimaging system can be configured to characterize the tissue type of thetissue being visualized based upon the particular combination ofconstituent materials.

FIG. 10 shows a graph 300 depicting how the absorption coefficient ofvarious biological materials varies across the EMR wavelength spectrum.In the graph 300, the vertical axis 302 represents absorptioncoefficient of the biological material in cm⁻¹, and the horizontal axis304 represents EMR wavelength in µm. A first line 306 in the graph 300represents the absorption coefficient of water at various EMRwavelengths, a second line 308 represents the absorption coefficient ofprotein at various EMR wavelengths, a third line 310 represents theabsorption coefficient of melanin at various EMR wavelengths, a fourthline 312 represents the absorption coefficient of deoxygenatedhemoglobin at various EMR wavelengths, a fifth line 314 represents theabsorption coefficient of oxygenated hemoglobin at various EMRwavelengths, and a sixth line 316 represents the absorption coefficientof collagen at various EMR wavelengths. Different tissue types havedifferent combinations of constituent materials and, therefore, thetissue type(s) being visualized by a surgical visualization system canbe identified and differentiated between according to the particularcombination of detected constituent materials. Accordingly, a spectralimaging system of a surgical visualization system can be configured toemit EMR at a number of different wavelengths, determine the constituentmaterials of the tissue based on the detected absorption EMR absorptionresponse at the different wavelengths, and then characterize the tissuetype based on the particular detected combination of constituentmaterials.

FIG. 11 shows an embodiment of the utilization of spectral imagingtechniques to visualize different tissue types and/or anatomicalstructures. In FIG. 11 , a spectral emitter 320 (e.g., the spectrallight source 150 of FIG. 4 ) is being utilized by an imaging system tovisualize a surgical site 322. The EMR emitted by the spectral emitter320 and reflected from the tissues and/or structures at the surgicalsite 322 is received by an image sensor (e.g., the image sensor 135 ofFIG. 4 ) to visualize the tissues and/or structures, which can be eithervisible (e.g., be located at a surface of the surgical site 322) orobscured (e.g., underlay other tissue and/or structures at the surgicalsite 322). In this embodiment, an imaging system (e.g., the imagingsystem 142 of FIG. 4 ) visualizes a tumor 324, an artery 326, andvarious abnormalities 328 (e.g., tissues not confirming to known orexpected spectral signatures) based upon the spectral signaturescharacterized by the differing absorptive characteristics (e.g.,absorption coefficient) of the constituent materials for each of thedifferent tissue/structure types. The visualized tissues and structurescan be displayed on a display screen associated with or coupled to theimaging system (e.g., the display 146 of the imaging system 142 of FIG.4 ), on a primary display (e.g., the primary display 819 of FIG. 19 ),on a non-sterile display (e.g., the non-sterile displays 807, 809 ofFIG. 19 ), on a display of a surgical hub (e.g., the display of thesurgical hub 806 of FIG. 19 ), on a device/instrument display, and/or onanother display.

The imaging system can be configured to tailor or update the displayedsurgical site visualization according to the identified tissue and/orstructure types. For example, as shown in FIG. 11 , the imaging systemcan display a margin 330 associated with the tumor 324 being visualizedon a display screen associated with or coupled to the imaging system, ona primary display, on a non-sterile display, on a display of a surgicalhub, on a device/instrument display, and/or on another display. Themargin 330 can indicate the area or amount of tissue that should beexcised to ensure complete removal of the tumor 324. A size of themargin 330 can be, for example, in a range of about 5 mm to about 10 mm.The surgical visualization system’s control system (e.g., the controlsystem 133 of FIG. 4 ) can be configured to control or update thedimensions of the margin 330 based on the tissues and/or structuresidentified by the imaging system. In this illustrated embodiment, theimaging system has identified multiple abnormalities 328 within thefield of view (FOV). Accordingly, the control system can adjust thedisplayed margin 330 to a first updated margin 332 having sufficientdimensions to encompass the abnormalities 328. Further, the imagingsystem has also identified the artery 326 partially overlapping with theinitially displayed margin 330 (as indicated by a highlighted region 334of the artery 326). Accordingly, the control system can adjust thedisplayed margin to a second updated margin 336 having sufficientdimensions to encompass the relevant portion of the artery 326.

Tissues and/or structures can also be imaged or characterized accordingto their reflective characteristics, in addition to or in lieu of theirabsorptive characteristics described above with respect to FIG. 10 andFIG. 11 , across the EMR wavelength spectrum. For example, FIG. 12 ,FIG. 13 , and FIG. 14 illustrate various graphs of reflectance ofdifferent types of tissues or structures across different EMRwavelengths. FIG. 12 is a graphical representation 340 of anillustrative ureter signature versus obscurants. FIG. 13 is a graphicalrepresentation 342 of an illustrative artery signature versusobscurants. FIG. 14 is a graphical representation 344 of an illustrativenerve signature versus obscurants. The plots in FIG. 12 , FIG. 13 , andFIG. 14 represent reflectance as a function of wavelength (nm) for theparticular structures (ureter, artery, and nerve) relative to thecorresponding reflectances of fat, lung tissue, and blood at thecorresponding wavelengths. These graphs are simply for illustrativepurposes and it should be understood that other tissues and/orstructures could have corresponding detectable reflectance signaturesthat would allow the tissues and/or structures to be identified andvisualized.

Select wavelengths for spectral imaging can be identified and utilizedbased on the anticipated critical structures and/or obscurants at asurgical site (e.g., “selective spectral” imaging). By utilizingselective spectral imaging, the amount of time required to obtain thespectral image can be minimized such that the information can beobtained in real-time and utilized intraoperatively. The wavelengths canbe selected by a medical practitioner or by a control circuit based oninput by a user, e.g., a medical practitioner. In certain instances, thewavelengths can be selected based on machine learning and/or big dataaccessible to the control circuit via, e.g., a cloud or surgical hub.

FIG. 15 illustrates one embodiment of spectral imaging to tissue beingutilized intraoperatively to measure a distance between a waveformemitter and a critical structure that is obscured by tissue. FIG. 15shows an embodiment of a time-of-flight sensor system 404 utilizingwaveforms 424, 425. The time-of-flight sensor system 404 can beincorporated into a surgical visualization system, e.g., as the sensorsystem 104 of the surgical visualization system 100 of FIG. 1 . Thetime-of-flight sensor system 404 includes a waveform emitter 406 and awaveform receiver 408 on the same surgical device 402 (e.g., the emitter106 and the receiver 108 on the same surgical device 102 of FIG. 1 ).The emitted wave 400 extends to a critical structure 401 (e.g., thecritical structure 101 of FIG. 1 ) from the emitter 406, and thereceived wave 425 is reflected back to by the receiver 408 from thecritical structure 401. The surgical device 402 in this illustratedembodiment is positioned through a trocar 410 that extends into a cavity407 in a patient. Although the trocar 410 is used in this in thisillustrated embodiment, other trocars or other access devices can beused, or no access device may be used.

The waveforms 424, 425 are configured to penetrate obscuring tissue 403,such as by having wavelengths in the NIR or SWIR spectrum ofwavelengths. A spectral signal (e.g., hyperspectral, multispectral, orselective spectral) or a photoacoustic signal is emitted from theemitter 406, as shown by a first arrow 407 pointing distally, and canpenetrate the tissue 403 in which the critical structure 401 isconcealed. The emitted waveform 424 is reflected by the criticalstructure 401, as shown by a second arrow 409 pointing proximally. Thereceived waveform 425 can be delayed due to a distance d between adistal end of the surgical device 402 and the critical structure 401.The waveforms 424, 425 can be selected to target the critical structure401 within the tissue 403 based on the spectral signature of thecritical structure 401, as described herein. The emitter 406 isconfigured to provide a binary signal on and off, as shown in FIG. 16 ,for example, which can be measured by the receiver 408.

Based on the delay between the emitted wave 424 and the received wave425, the time-of-flight sensor system 404 is configured to determine thedistance d. A time-of-flight timing diagram 430 for the emitter 406 andthe receiver 408 of FIG. 15 is shown in FIG. 16 . The delay is afunction of the distance d and the distance d is given by:

$d = \frac{ct}{2} \bullet \frac{q_{2}}{q_{1} + q_{2}}$

where c = the speed of light; t = length of pulse; q₁ = accumulatedcharge while light is emitted; and q₂ = accumulated charge while lightis not being emitted.

The time-of-flight of the waveforms 424, 425 corresponds to the distanced in FIG. 15 . In various instances, additional emitters/receiversand/or pulsing signals from the emitter 406 can be configured to emit anon-penetrating signal. The non-penetrating signal can be configured todetermine the distance from the emitter 406 to the surface 405 of theobscuring tissue 403. In various instances, a depth of the criticalstructure 401 can be determined by:

d_(A) = d_(w)- d_(t)

where d_(A) = the depth of the critical structure 401; d_(w) = thedistance from the emitter 406 to the critical structure 401 (d in FIG.15 ); and d_(t) _(,)= the distance from the emitter 406 (on the distalend of the surgical device 402) to the surface 405 of the obscuringtissue 403.

FIG. 17 illustrates another embodiment of a time-of-flight sensor system504 utilizing waves 524 a, 524 b, 524 c, 525 a, 525 b, 525 c is shown.The time-of-flight sensor system 504 can be incorporated into a surgicalvisualization system, e.g., as the sensor system 104 of the surgicalvisualization system 100 of FIG. 1 . The time-of-flight sensor system504 includes a waveform emitter 506 and a waveform receiver 508 (e.g.,the emitter 106 and the receiver 108 of FIG. 1 ). The waveform emitter506 is positioned on a first surgical device 502 a (e.g., the surgicaldevice 102 of FIG. 1 ), and the waveform receiver 508 is positioned on asecond surgical device 502 b. The surgical devices 502 a, 502 b arepositioned through first and second trocars 510 a, 510 b, respectively,which extend into a cavity 507 in a patient. Although the trocars 510 a,510 b are used in this in this illustrated embodiment, other trocars orother access devices can be used, or no access device may be used. Theemitted waves 524 a, 524 b, 524 c extend toward a surgical site from theemitter 506, and the received waves 525 a, 525 b, 525 c are reflectedback to the receiver 508 from various structures and/or surfaces at thesurgical site.

The different emitted waves 524 a, 524 b, 524 c are configured to targetdifferent types of material at the surgical site. For example, the wave524 a targets obscuring tissue 503, the wave 524 b targets a firstcritical structure 501 a (e.g., the critical structure 101 of FIG. 1 ),which is a vessel in this illustrated embodiment, and the wave 524 ctargets a second critical structure 501 b (e.g., the critical structure101 of FIG. 1 ), which is a cancerous tumor in this illustratedembodiment. The wavelengths of the waves 524 a, 524 b, 524 c can be inthe visible light, NIR, or SWIR spectrum of wavelengths. For example,visible light can be reflected off a surface 505 of the tissue 503, andNIR and/or SWIR waveforms can penetrate the surface 505 of the tissue503. In various aspects, as described herein, a spectral signal (e.g.,hyperspectral, multispectral, or selective spectral) or a photoacousticsignal can be emitted from the emitter 506. The waves 524 b, 524 c canbe selected to target the critical structures 501 a, 501 b within thetissue 503 based on the spectral signature of the critical structures501 a, 501 b, as described herein. Photoacoustic imaging is furtherdescribed in various U.S. patent applications, which are incorporated byreference herein in the present disclosure.

The emitted waves 524 a, 524 b, 524 c are reflected off the targetedmaterial, namely the surface 505, the first critical structure 501 a,and the second structure 501 b, respectively. The received waveforms 525a, 525 b, 525 c can be delayed due to distances d_(1a), d_(2a), d_(3a),d_(1b), d_(2b), d_(2c).

In the time-of-flight sensor system 504, in which the emitter 506 andthe receiver 508 are independently positionable (e.g., on separatesurgical devices 502 a, 502 b and/or controlled by separate roboticarms), the various distances d_(1a), d_(2a), d_(3a), d_(1b), d_(2b),d_(2c) can be calculated from the known position of the emitter 506 andthe receiver 508. For example, the positions can be known when thesurgical devices 502 a, 502 b are robotically-controlled. Knowledge ofthe positions of the emitter 506 and the receiver 508, as well as thetime of the photon stream to target a certain tissue and the informationreceived by the receiver 508 of that particular response can allow adetermination of the distances d_(1a), d_(2a), d_(3a), d_(1b), d_(2b),d_(2c). In one aspect, the distance to the obscured critical structures501 a, 501 b can be triangulated using penetrating wavelengths. Becausethe speed of light is constant for any wavelength of visible orinvisible light, the time-of-flight sensor system 504 can determine thevarious distances.

In a view provided to the medical practitioner, such as on a display,the receiver 508 can be rotated such that a center of mass of the targetstructure in the resulting images remains constant, e.g., in a planeperpendicular to an axis of a select target structure 503, 501 a, or 501b. Such an orientation can quickly communicate one or more relevantdistances and/or perspectives with respect to the target structure. Forexample, as shown in FIG. 17 , the surgical site is displayed from aviewpoint in which the critical structure 501 a is perpendicular to theviewing plane (e.g., the vessel is oriented in/out of the page). Such anorientation can be default setting; however, the view can be rotated orotherwise adjusted by a medical practitioner. In certain instances, themedical practitioner can toggle between different surfaces and/or targetstructures that define the viewpoint of the surgical site provided bythe imaging system.

As in this illustrated embodiment, the receiver 508 can be mounted onthe trocar 510 b (or other access device) through which the surgicaldevice 502 b is positioned. In other embodiments, the receiver 508 canbe mounted on a separate robotic arm for which the three-dimensionalposition is known. In various instances, the receiver 508 can be mountedon a movable arm that is separate from a robotic surgical system thatcontrols the surgical device 502 a or can be mounted to an operatingroom (OR) table or fixture that is intraoperatively registerable to therobot coordinate plane. In such instances, the position of the emitter506 and the receiver 508 can be registerable to the same coordinateplane such that the distances can be triangulated from outputs from thetime-of-flight sensor system 504.

Combining time-of-flight sensor systems and near-infrared spectroscopy(NIRS), termed TOF-NIRS, which is capable of measuring the time-resolvedprofiles of NIR light with nanosecond resolution can be found in“Time-Of-Flight Near-Infrared Spectroscopy For NondestructiveMeasurement Of Internal Quality In Grapefruit,” Journal of the AmericanSociety for Horticultural Science, May 2013 vol. 138 no. 3 225-228,which is hereby incorporated by reference in its entirety.

Embodiments of visualization systems and aspects and uses thereof aredescribed further in U.S. Pat. Pub. No. 2020/0015923 entitled “SurgicalVisualization Platform” filed Sep. 11, 2018, U.S. Pat. Pub. No.2020/0015900 entitled “Controlling An Emitter Assembly Pulse Sequence”filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015668 entitled “SingularEMR Source Emitter Assembly” filed Sep. 11, 2018, U.S. Pat. Pub. No.2020/0015925 entitled “Combination Emitter And Camera Assembly” filedSep. 11, 2018, U.S. Pat. Pub. No. 2020/0015899 entitled “SurgicalVisualization With Proximity Tracking Features” filed Sep. 11, 2018,U.S. Pat. Pub. No. 2020/0015903 entitled “Surgical Visualization OfMultiple Targets” filed Sep. 11, 2018, U.S. Pat. No. 10,792,034 entitled“Visualization Of Surgical Devices” filed Sep. 11, 2018, U.S. Pat. Pub.No. 2020/0015897 entitled “Operative Communication Of Light” filed Sep.11, 2018, U.S. Pat. Pub. No. 2020/0015924 entitled “Robotic LightProjection Tools” filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015898entitled “Surgical Visualization Feedback System” filed Sep. 11, 2018,U.S. Pat. Pub. No. 2020/0015906 entitled “Surgical Visualization AndMonitoring” filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015907entitled “Integration Of Imaging Data” filed Sep. 11, 2018, U.S. Pat.No. 10,925,598 entitled “Robotically-Assisted Surgical Suturing Systems”filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015901 entitled “SafetyLogic For Surgical Suturing Systems” filed Sep. 11, 2018, U.S. Pat. Pub.No. 2020/0015914 entitled “Robotic Systems With Separate PhotoacousticReceivers” filed Sep. 11, 2018, U.S. Pat. Pub. No. 2020/0015902 entitled“Force Sensor Through Structured Light Deflection” filed Sep. 11, 2018,U.S. Pat. Pub. No. 2019/0201136 entitled “Method Of Hub Communication”filed Dec. 4, 2018, U.S. Pat. App. No. 16/729,772 entitled “AnalyzingSurgical Trends By A Surgical System” filed Dec. 30, 2019, U.S. Pat.App. No. 16/729,747 entitled “Dynamic Surgical Visualization Systems”filed Dec. 30, 2019, U.S. Pat. App. No. 16/729,744 entitled“Visualization Systems Using Structured Light” filed Dec. 30, 2019, U.S.Pat. App. No. 16/729,778 entitled “System And Method For Determining,Adjusting, And Managing Resection Margin About A Subject Tissue” filedDec. 30, 2019, U.S. Pat. App. No. 16/729,729 entitled “Surgical SystemsFor Proposing And Corroborating Organ Portion Removals” filed Dec. 30,2019, U.S. Pat. App. No. 16/729,778 entitled “Surgical System ForOverlaying Surgical Instrument Data Onto A Virtual Three DimensionalConstruct Of An Organ” filed Dec. 30, 2019, U.S. Pat. App. No.16/729,751 entitled “Surgical Systems For Generating Three DimensionalConstructs Of Anatomical Organs And Coupling Identified AnatomicalStructures Thereto” filed Dec. 30, 2019, U.S. Pat. App. No. 16/729,740entitled “Surgical Systems Correlating Visualization Data And PoweredSurgical Instrument Data” filed Dec. 30, 2019, U.S. Pat. App. No.16/729,737 entitled “Adaptive Surgical System Control According ToSurgical Smoke Cloud Characteristics” filed Dec. 30, 2019, U.S. Pat.App. No. 16/729,796 entitled “Adaptive Surgical System Control AccordingTo Surgical Smoke Particulate Characteristics” filed Dec. 30, 2019, U.S.Pat. App. No. 16/729,803 entitled “Adaptive Visualization By A SurgicalSystem” filed Dec. 30, 2019, U.S. Pat. App. No. 16/729,807 entitled“Method Of Using Imaging Devices In Surgery” filed Dec. 30, 2019, U.S.Pat. App No. 17/493,913 entitled “Surgical Methods Using FiducialIdentification And Tracking” filed on Oct. 5, 2021, U.S. Pat. App No.17/494,364 entitled “Surgical Methods For Control Of One VisualizationWith Another” filed on Oct. 5, 2021, U.S. Pat. App No. 17/450,020entitled “Methods And Systems For Controlling Cooperative SurgicalInstruments” filed on Oct. 5, 2021, U.S. Pat. App No. 17/450,025entitled “Methods And Systems For Controlling Cooperative SurgicalInstruments With Variable Surgical Site Access Trajectories” filed onOct. 5, 2021, U.S. Pat. App No. 17/450,027 entitled “Methods And SystemsFor Controlling Cooperative Surgical Instruments” filed on Oct. 5, 2021,and U.S. Pat. App No. 17/449,765 entitled “Cooperative Access HybridProcedures” filed Oct. 1, 2021, which are hereby incorporated byreference in their entireties.

Surgical Hubs

The various visualization or imaging systems described herein can beincorporated into a system that includes a surgical hub. In general, asurgical hub can be a component of a comprehensive digital medicalsystem capable of spanning multiple medical facilities and configured toprovide integrated and comprehensive improved medical care to a vastnumber of patients. The comprehensive digital medical system includes acloud-based medical analytics system that is configured to interconnectto multiple surgical hubs located across many different medicalfacilities. The surgical hubs are configured to interconnect with one ormore elements, such as one or more surgical instruments that are used toconduct medical procedures on patients and/or one or more visualizationsystems that are used during performance of medical procedures. Thesurgical hubs provide a wide array of functionality to improve theoutcomes of medical procedures. The data generated by the varioussurgical devices, visualization systems, and surgical hubs about thepatient and the medical procedure may be transmitted to the cloud-basedmedical analytics system. This data may then be aggregated with similardata gathered from many other surgical hubs, visualization systems, andsurgical instruments located at other medical facilities. Variouspatterns and correlations may be found through the cloud-based analyticssystem analyzing the collected data. Improvements in the techniques usedto generate the data may be generated as a result, and theseimprovements may then be disseminated to the various surgical hubs,visualization systems, and surgical instruments. Due to theinterconnectedness of all of the aforementioned components, improvementsin medical procedures and practices may be found that otherwise may notbe found if the many components were not so interconnected.

Examples of surgical hubs configured to receive, analyze, and outputdata, and methods of using such surgical hubs, are further described inU.S. Pat. Pub. No. 2019/0200844 entitled “Method Of Hub Communication,Processing, Storage And Display” filed Dec. 4, 2018, U.S. Pat. Pub. No.2019/0200981 entitled “Method Of Compressing Tissue Within A StaplingDevice And Simultaneously Displaying The Location Of The Tissue WithinThe Jaws” filed Dec. 4, 2018, U.S. Pat. Pub. No. 2019/0201046 entitled“Method For Controlling Smart Energy Devices” filed Dec. 4, 2018, U.S.Pat. Pub. No. 2019/0201114 entitled “Adaptive Control Program UpdatesFor Surgical Hubs” filed Mar. 29, 2018,U.S. Pat. Pub. No. 2019/0201140entitled “Surgical Hub Situational Awareness” filed Mar. 29, 2018, U.S.Pat. Pub. No. 2019/0206004 entitled “Interactive Surgical Systems WithCondition Handling Of Devices And Data Capabilities” filed Mar. 29,2018, U.S. Pat. Pub. No. 2019/0206555 entitled “Cloud-based MedicalAnalytics For Customization And Recommendations To A User” filed Mar.29, 2018, and U.S. Pat. Pub. No. 2019/0207857 entitled “Surgical NetworkDetermination Of Prioritization Of Communication, Interaction, OrProcessing Based On System Or Device Needs” filed Nov. 6, 2018, whichare hereby incorporated by reference in their entireties.

FIG. 18 illustrates one embodiment of a computer-implemented interactivesurgical system 700 that includes one or more surgical systems 702 and acloud-based system (e.g., a cloud 704 that can include a remote server713 coupled to a storage device 705). Each surgical system 702 includesat least one surgical hub 706 in communication with the cloud 704. Inone example, as illustrated in FIG. 18 , the surgical system 702includes a visualization system 708, a robotic system 710, and anintelligent (or “smart”) surgical instrument 712, which are configuredto communicate with one another and/or the hub 706. The intelligentsurgical instrument 712 can include imaging device(s). The surgicalsystem 702 can include an M number of hubs 706, an N number ofvisualization systems 708, an O number of robotic systems 710, and a Pnumber of intelligent surgical instruments 712, where M, N, O, and P areintegers greater than or equal to one that may or may not be equal toany one or more of each other. Various exemplary intelligent surgicalinstruments and robotic systems are described herein.

Data received by a surgical hub from a surgical visualization system canbe used in any of a variety of ways. In an exemplary embodiment, thesurgical hub can receive data from a surgical visualization system inuse with a patient in a surgical setting, e.g., in use in an operatingroom during performance of a surgical procedure. The surgical hub canuse the received data in any of one or more ways, as discussed herein.

The surgical hub can be configured to analyze received data in real timewith use of the surgical visualization system and adjust control one ormore of the surgical visualization system and/or one or more intelligentsurgical instruments in use with the patient based on the analysis ofthe received data. Such adjustment can include, for example, adjustingone or operational control parameters of intelligent surgicalinstrument(s), causing one or more sensors of one or more intelligentsurgical instruments to take a measurement to help gain an understandingof the patient’s current physiological condition, and/or currentoperational status of an intelligent surgical instrument, and otheradjustments. Controlling and adjusting operation of intelligent surgicalinstruments is discussed further below. Examples of operational controlparameters of an intelligent surgical instrument include motor speed,cutting element speed, time, duration, level of energy application, andlight emission. Examples of surgical hubs and of controlling andadjusting intelligent surgical instrument operation are describedfurther in previously mentioned U.S. Pat. App. No. 16/729,772 entitled“Analyzing Surgical Trends By A Surgical System” filed Dec. 30, 2019,U.S. Pat. App. No. 16/729,747 entitled “Dynamic Surgical VisualizationSystems” filed Dec. 30, 2019, U.S. Pat. App. No. 16/729,744 entitled“Visualization Systems Using Structured Light” filed Dec. 30, 2019, U.S.Pat. App. No. 16/729,778 entitled “System And Method For Determining,Adjusting, And Managing Resection Margin About A Subject Tissue” filedDec. 30, 2019, U.S. Pat. App. No. 16/729,729 entitled “Surgical SystemsFor Proposing And Corroborating Organ Portion Removals” filed Dec. 30,2019, U.S. Pat. App. No. 16/729,778 entitled “Surgical System ForOverlaying Surgical Instrument Data Onto A Virtual Three DimensionalConstruct Of An Organ” filed Dec. 30, 2019, U.S. Pat. App. No.16/729,751 entitled “Surgical Systems For Generating Three DimensionalConstructs Of Anatomical Organs And Coupling Identified AnatomicalStructures Thereto” filed Dec. 30, 2019, U.S. Pat. App. No. 16/729,740entitled “Surgical Systems Correlating Visualization Data And PoweredSurgical Instrument Data” filed Dec. 30, 2019, U.S. Pat. App. No.16/729,737 entitled “Adaptive Surgical System Control According ToSurgical Smoke Cloud Characteristics” filed Dec. 30, 2019, U.S. Pat.App. No. 16/729,796 entitled “Adaptive Surgical System Control AccordingTo Surgical Smoke Particulate Characteristics” filed Dec. 30, 2019, U.S.Pat. App. No. 16/729,803 entitled “Adaptive Visualization By A SurgicalSystem” filed Dec. 30, 2019, and U.S. Pat. App. No. 16/729,807 entitled“Method Of Using Imaging Devices In Surgery” filed Dec. 30, 2019, and inU.S. Pat. App. No. 17/068,857 entitled “Adaptive Responses From SmartPackaging Of Drug Delivery Absorbable Adjuncts” filed Oct. 13, 2020,U.S. Pat. App. No. 17/068,858 entitled “Drug Administration Devices ThatCommunicate With Surgical Hubs” filed Oct. 13, 2020, U.S. Pat. App. No.17/068,859 entitled “Controlling Operation Of Drug AdministrationDevices Using Surgical Hubs” filed Oct. 13, 2020, U.S. Pat. App. No.17/068,863 entitled “Patient Monitoring Using Drug AdministrationDevices” filed Oct. 13, 2020, U.S. Pat. App. No. 17/068,865 entitled“Monitoring And Communicating Information Using Drug AdministrationDevices” filed Oct. 13, 2020, and U.S. Pat. App. No. 17/068,867 entitled“Aggregating And Analyzing Drug Administration Data” filed Oct. 13,2020, which are hereby incorporated by reference in their entireties.

The surgical hub can be configured to cause visualization of thereceived data to be provided in the surgical setting on a display sothat a medical practitioner in the surgical setting can view the dataand thereby receive an understanding of the operation of the imagingdevice(s) in use in the surgical setting. Such information provided viavisualization can include text and/or images.

FIG. 19 illustrates one embodiment of a surgical system 802 including asurgical hub 806 (e.g., the surgical hub 706 of FIG. 18 or othersurgical hub described herein), a robotic surgical system 810 (e.g., therobotic surgical system 110 of FIG. 1 or other robotic surgical systemherein), and a visualization system 808 (e.g., the visualization system100 of FIG. 1 or other visualization system described herein). Thesurgical hub 806 can be in communication with a cloud, as discussedherein. FIG. 19 shows the surgical system 802 being used to perform asurgical procedure on a patient who is lying down on an operating table814 in a surgical operating room 816. The robotic system 810 includes asurgeon’s console 818, a patient side cart 820 (surgical robot), and arobotic system surgical hub 822. The robotic system surgical hub 822 isgenerally configured similar to the surgical hub 822 and can be incommunication with a cloud. In some embodiments, the robotic systemsurgical hub 822 and the surgical hub 806 can be combined. The patientside cart 820 can manipulate an intelligent surgical tool 812 through aminimally invasive incision in the body of the patient while a medicalpractitioner, e.g., a surgeon, nurse, and/or other medical practitioner,views the surgical site through the surgeon’s console 818. An image ofthe surgical site can be obtained by an imaging device 824 (e.g., theimaging device 120 of FIG. 1 or other imaging device described herein),which can be manipulated by the patient side cart 820 to orient theimaging device 824. The robotic system surgical hub 822 can be used toprocess the images of the surgical site for subsequent display to thesurgeon through the surgeon’s console 818.

A primary display 819 is positioned in the sterile field of theoperating room 816 and is configured to be visible to an operator at theoperating table 814. In addition, as in this illustrated embodiment, avisualization tower 811 can positioned outside the sterile field. Thevisualization tower 811 includes a first non-sterile display 807 and asecond non-sterile display 809, which face away from each other. Thevisualization system 808, guided by the surgical hub 806, is configuredto utilize the displays 807, 809, 819 to coordinate information flow tomedical practitioners inside and outside the sterile field. For example,the surgical hub 806 can cause the visualization system 808 to display asnapshot and/or a video of a surgical site, as obtained by the imagingdevice 824, on one or both of the non-sterile displays 807, 809, whilemaintaining a live feed of the surgical site on the primary display 819.The snapshot and/or video on the non-sterile display 807 and/or 809 canpermit a non-sterile medical practitioner to perform a diagnostic steprelevant to the surgical procedure, for example.

The surgical hub 806 is configured to route a diagnostic input orfeedback entered by a non-sterile medical practitioner at thevisualization tower 811 to the primary display 819 within the sterilefield, where it can be viewed by a sterile medical practitioner at theoperating table 814. For example, the input can be in the form of amodification to the snapshot and/or video displayed on the non-steriledisplay 807 and/or 809, which can be routed to the primary display 819by the surgical hub 806.

The surgical hub 806 is configured to coordinate information flow to adisplay of the intelligent surgical instrument 812, as is described invarious U.S. Patent Applications that are incorporated by referenceherein in the present disclosure. A diagnostic input or feedback enteredby a non-sterile operator at the visualization tower 818 can be routedby the surgical hub 806 to the display 819 within the sterile field,where it can be viewed by the operator of the surgical instrument 812and/or by other medical practitioner(s) in the sterile field.

The intelligent surgical instrument 812 and the imaging device 824,which is also an intelligent surgical tool, is being used with thepatient in the surgical procedure as part of the surgical system 802.Other intelligent surgical instruments 812 a that can be used in thesurgical procedure, e.g., that can be removably coupled to the patientside cart 820 and be in communication with the robotic surgical system810 and the surgical hub 806, are also shown in FIG. 19 as beingavailable. Non-intelligent (or “dumb”) surgical instruments 817, e.g.,scissors, trocars, cannulas, scalpels, etc., that cannot be incommunication with the robotic surgical system 810 and the surgical hub806 are also shown in FIG. 19 as being available for use.

Operating Intelligent Surgical Instruments

An intelligent surgical device can have an algorithm stored thereon,e.g., in a memory thereof, configured to be executable on board theintelligent surgical device, e.g., by a processor thereof, to controloperation of the intelligent surgical device. In some embodiments,instead of or in addition to being stored on the intelligent surgicaldevice, the algorithm can be stored on a surgical hub, e.g., in a memorythereof, that is configured to communicate with the intelligent surgicaldevice.

The algorithm is stored in the form of one or more sets of pluralitiesof data points defining and/or representing instructions, notifications,signals, etc. to control functions of the intelligent surgical device.In some embodiments, data gathered by the intelligent surgical devicecan be used by the intelligent surgical device, e.g., by a processor ofthe intelligent surgical device, to change at least one variableparameter of the algorithm. As discussed above, a surgical hub can be incommunication with an intelligent surgical device, so data gathered bythe intelligent surgical device can be communicated to the surgical huband/or data gathered by another device in communication with thesurgical hub can be communicated to the surgical hub, and data can becommunicated from the surgical hub to the intelligent surgical device.Thus, instead of or in addition to the intelligent surgical device beingconfigured to change a stored variable parameter, the surgical hub canbe configured to communicate the changed at least one variable, alone oras part of the algorithm, to the intelligent surgical device and/or thesurgical hub can communicate an instruction to the intelligent surgicaldevice to change the at least one variable as determined by the surgicalhub.

The at least one variable parameter is among the algorithm’s datapoints, e.g., are included in instructions for operating the intelligentsurgical device, and are thus each able to be changed by changing one ormore of the stored pluralities of data points of the algorithm. Afterthe at least one variable parameter has been changed, subsequentexecution of the algorithm is according to the changed algorithm. Assuch, operation of the intelligent surgical device over time can bemanaged for a patient to increase the beneficial results use of theintelligent surgical device by taking into consideration actualsituations of the patient and actual conditions and/or results of thesurgical procedure in which the intelligent surgical device is beingused. Changing the at least one variable parameter is automated toimprove patient outcomes. Thus, the intelligent surgical device can beconfigured to provide personalized medicine based on the patient and thepatient’s surrounding conditions to provide a smart system. In asurgical setting in which the intelligent surgical device is being usedduring performance of a surgical procedure, automated changing of the atleast one variable parameter may allow for the intelligent surgicaldevice to be controlled based on data gathered during the performance ofthe surgical procedure, which may help ensure that the intelligentsurgical device is used efficiently and correctly and/or may help reducechances of patient harm by harming a critical anatomical structure.

The at least one variable parameter can be any of a variety of differentoperational parameters. Examples of variable parameters include motorspeed, motor torque, energy level, energy application duration, tissuecompression rate, jaw closure rate, cutting element speed, loadthreshold, etc.

FIG. 20 illustrates one embodiment of an intelligent surgical instrument900 including a memory 902 having an algorithm 904 stored therein thatincludes at least one variable parameter. The algorithm 904 can be asingle algorithm or can include a plurality of algorithms, e.g.,separate algorithms for different aspects of the surgical instrument’soperation, where each algorithm includes at least one variableparameter. The intelligent surgical instrument 900 can be the surgicaldevice 102 of FIG. 1 , the imaging device 120 of FIG. 1 , the surgicaldevice 202 of FIG. 8 , the imaging device 220 of FIG. 8 , the surgicaldevice 402 of FIG. 15 , the surgical device 502 a of FIG. 17 , thesurgical device 502 b of FIG. 17 , the surgical device 712 of FIG. 18 ,the surgical device 812 of FIG. 19 , the imaging device 824 of FIG. 19 ,or other intelligent surgical instrument. The surgical instrument 900also includes a processor 906 configured to execute the algorithm 904 tocontrol operation of at least one aspect of the surgical instrument 900.To execute the algorithm 904, the processor 906 is configured to run aprogram stored in the memory 902 to access a plurality of data points ofthe algorithm 904 in the memory 902.

The surgical instrument 900 also includes a communications interface908, e.g., a wireless transceiver or other wired or wirelesscommunications interface, configured to communicate with another device,such as a surgical hub 910. The communications interface 908 can beconfigured to allow one-way communication, such as providing data to aremote server (e.g., a cloud server or other server) and/or to a local,surgical hub server, and/or receiving instructions or commands from aremote server and/or a local, surgical hub server, or two-waycommunication, such as providing information, messages, data, etc.regarding the surgical instrument 900 and/or data stored thereon andreceiving instructions, such as from a doctor; a remote server regardingupdates to software; a local, surgical hub server regarding updates tosoftware; etc.

The surgical instrument 900 is simplified in FIG. 20 and can includeadditional components, e.g., a bus system, a handle, a elongate shafthaving an end effector at a distal end thereof, a power source, etc. Theprocessor 906 can also be configured to execute instructions stored inthe memory 902 to control the device 900 generally, including otherelectrical components thereof such as the communications interface 908,an audio speaker, a user interface, etc.

The processor 906 is configured to change at least one variableparameter of the algorithm 904 such that a subsequent execution of thealgorithm 904 will be in accordance with the changed at least onevariable parameter. To change the at least one variable parameter of thealgorithm 904, the processor 906 is configured to modify or update thedata point(s) of the at least one variable parameter in the memory 902.The processor 906 can be configured to change the at least one variableparameter of the algorithm 904 in real time with use of the surgicaldevice 900 during performance of a surgical procedure, which mayaccommodate real time conditions.

Additionally or alternatively to the processor 906 changing the at leastone variable parameter, the processor 906 can be configured to changethe algorithm 904 and/or at least one variable parameter of thealgorithm 904 in response to an instruction received from the surgicalhub 910. In some embodiments, the processor 906 is configured to changethe at least one variable parameter only after communicating with thesurgical hub 910 and receiving an instruction therefrom, which may helpensure coordinated action of the surgical instrument 900 with otheraspects of the surgical procedure in which the surgical instrument 900is being used.

In an exemplary embodiment, the processor 906 executes the algorithm 904to control operation of the surgical instrument 900, changes the atleast one variable parameter of the algorithm 904 based on real timedata, and executes the algorithm 904 after changing the at least onevariable parameter to control operation of the surgical instrument 900.

FIG. 21 illustrates one embodiment of a method 912 of using of thesurgical instrument 900 including a change of at least one variableparameter of the algorithm 904. The processor 906 controls 914 operationof the surgical instrument 900 by executing the algorithm 904 stored inthe memory 902. Based on any of this subsequently known data and/orsubsequently gathered data, the processor 904 changes 916 the at leastone variable parameter of the algorithm 904 as discussed above. Afterchanging the at least one variable parameter, the processor 906 controls918 operation of the surgical instrument 900 by executing the algorithm904, now with the changed at least one variable parameter. The processor904 can change 916 the at least one variable parameter any number oftimes during performance of a surgical procedure, e.g., zero, one, two,three, etc. During any part of the method 912, the surgical instrument900 can communicate with one or more computer systems, e.g., thesurgical hub 910, a remote server such as a cloud server, etc., usingthe communications interface 908 to provide data thereto and/or receiveinstructions therefrom.

Situational Awareness

Operation of an intelligent surgical instrument can be altered based onsituational awareness of the patient. The operation of the intelligentsurgical instrument can be altered manually, such as by a user of theintelligent surgical instrument handling the instrument differently,providing a different input to the instrument, ceasing use of theinstrument, etc. Additionally or alternatively, the operation of anintelligent surgical instrument can be changed automatically by analgorithm of the instrument being changed, e.g., by changing at leastone variable parameter of the algorithm. As mentioned above, thealgorithm can be adjusted automatically without user input requestingthe change. Automating the adjustment during performance of a surgicalprocedure may help save time, may allow medical practitioners to focuson other aspects of the surgical procedure, and/or may ease the processof using the surgical instrument for a medical practitioner, which eachmay improve patient outcomes, such as by avoiding a critical structure,controlling the surgical instrument with consideration of a tissue typethe instrument is being used on and/or near, etc.

The visualization systems described herein can be utilized as part of asituational awareness system that can be embodied or executed by asurgical hub, e.g., the surgical hub 706, the surgical hub 806, or othersurgical hub described herein. In particular, characterizing,identifying, and/or visualizing surgical instruments (including theirpositions, orientations, and actions), tissues, structures, users,and/or other things located within the surgical field or the operatingtheater can provide contextual data that can be utilized by asituational awareness system to infer various information, such as atype of surgical procedure or a step thereof being performed, a type oftissue(s) and/or structure(s) being manipulated by a surgeon or othermedical practitioner, and other information. The contextual data canthen be utilized by the situational awareness system to provide alertsto a user, suggest subsequent steps or actions for the user toundertake, prepare surgical devices in anticipation for their use (e.g.,activate an electrosurgical generator in anticipation of anelectrosurgical instrument being utilized in a subsequent step of thesurgical procedure, etc.), control operation of intelligent surgicalinstruments (e.g., customize surgical instrument operational parametersof an algorithm as discussed further below), and so on.

Although an intelligent surgical device including an algorithm thatresponds to sensed data, e.g., by having at least one variable parameterof the algorithm changed, can be an improvement over a “dumb” devicethat operates without accounting for sensed data, some sensed data canbe incomplete or inconclusive when considered in isolation, e.g.,without the context of the type of surgical procedure being performed orthe type of tissue that is being operated on. Without knowing theprocedural context (e.g., knowing the type of tissue being operated onor the type of procedure being performed), the algorithm may control thesurgical device incorrectly or sub-optimally given the particularcontext-free sensed data. For example, the optimal manner for analgorithm to control a surgical instrument in response to a particularsensed parameter can vary according to the particular tissue type beingoperated on. This is due to the fact that different tissue types havedifferent properties (e.g., resistance to tearing, ease of being cut,etc.) and thus respond differently to actions taken by surgicalinstruments. Therefore, it may be desirable for a surgical instrument totake different actions even when the same measurement for a particularparameter is sensed. As one example, the optimal manner in which tocontrol a surgical stapler in response to the surgical stapler sensingan unexpectedly high force to close its end effector will vary dependingupon whether the tissue type is susceptible or resistant to tearing. Fortissues that are susceptible to tearing, such as lung tissue, thesurgical instrument’s control algorithm would optimally ramp down themotor in response to an unexpectedly high force to close to avoidtearing the tissue, e.g., change a variable parameter controlling motorspeed or torque so the motor is slower. For tissues that are resistantto tearing, such as stomach tissue, the instrument’s algorithm wouldoptimally ramp up the motor in response to an unexpectedly high force toclose to ensure that the end effector is clamped properly on the tissue,e.g., change a variable parameter controlling motor speed or torque sothe motor is faster. Without knowing whether lung or stomach tissue hasbeen clamped, the algorithm may be sub-optimally changed or not changedat all.

A surgical hub can be configured to derive information about a surgicalprocedure being performed based on data received from various datasources and then control modular devices accordingly. In other words,the surgical hub can be configured to infer information about thesurgical procedure from received data and then control the modulardevices operably coupled to the surgical hub based upon the inferredcontext of the surgical procedure. Modular devices can include anysurgical device that is controllable by a situational awareness system,such as visualization system devices (e.g., a camera, a display screen,etc.), smart surgical instruments (e.g., an ultrasonic surgicalinstrument, an electrosurgical instrument, a surgical stapler, smokeevacuators, scopes, etc.). A modular device can include sensor(s)configured to detect parameters associated with a patient with which thedevice is being used and/or associated with the modular device itself.

The contextual information derived or inferred from the received datacan include, for example, a type of surgical procedure being performed,a particular step of the surgical procedure that the surgeon (or othermedical practitioner) is performing, a type of tissue being operated on,or a body cavity that is the subject of the surgical procedure. Thesituational awareness system of the surgical hub can be configured toderive the contextual information from the data received from the datasources in a variety of different ways. In an exemplary embodiment, thecontextual information received by the situational awareness system ofthe surgical hub is associated with a particular control adjustment orset of control adjustments for one or more modular devices. The controladjustments each correspond to a variable parameter. In one example, thesituational awareness system includes a pattern recognition system, ormachine learning system (e.g., an artificial neural network), that hasbeen trained on training data to correlate various inputs (e.g., datafrom databases, patient monitoring devices, and/or modular devices) tocorresponding contextual information regarding a surgical procedure. Inother words, a machine learning system can be trained to accuratelyderive contextual information regarding a surgical procedure from theprovided inputs. In another example, the situational awareness systemcan include a lookup table storing precharacterized contextualinformation regarding a surgical procedure in association with one ormore inputs (or ranges of inputs) corresponding to the contextualinformation. In response to a query with one or more inputs, the lookuptable can return the corresponding contextual information for thesituational awareness system for controlling at least one modulardevice. In another example, the situational awareness system includes afurther machine learning system, lookup table, or other such system,which generates or retrieves one or more control adjustments for one ormore modular devices when provided the contextual information as input.

A surgical hub including a situational awareness system may provide anynumber of benefits for a surgical system. One benefit includes improvingthe interpretation of sensed and collected data, which would in turnimprove the processing accuracy and/or the usage of the data during thecourse of a surgical procedure. Another benefit is that the situationalawareness system for the surgical hub may improve surgical procedureoutcomes by allowing for adjustment of surgical instruments (and othermodular devices) for the particular context of each surgical procedure(such as adjusting to different tissue types) and validating actionsduring a surgical procedure. Yet another benefit is that the situationalawareness system may improve surgeon’s and/or other medicalpractitioners’ efficiency in performing surgical procedures byautomatically suggesting next steps, providing data, and adjustingdisplays and other modular devices in the surgical theater according tothe specific context of the procedure. Another benefit includesproactively and automatically controlling modular devices according tothe particular step of the surgical procedure that is being performed toreduce the number of times that medical practitioners are required tointeract with or control the surgical system during the course of asurgical procedure, such as by a situationally aware surgical hubproactively activating a generator to which an RF electrosurgicalinstrument is connected if it determines that a subsequent step of theprocedure requires the use of the instrument. Proactively activating theenergy source allows the instrument to be ready for use a soon as thepreceding step of the procedure is completed.

For example, a situationally aware surgical hub can be configured todetermine what type of tissue is being operated on. Therefore, when anunexpectedly high force to close a surgical instrument’s end effector isdetected, the situationally aware surgical hub can be configured tocorrectly ramp up or ramp down a motor of the surgical instrument forthe type of tissue, e.g., by changing or causing change of at least onevariable parameter of an algorithm for the surgical instrument regardingmotor speed or torque.

For another example, a type of tissue being operated can affectadjustments that are made to compression rate and load thresholds of asurgical stapler for a particular tissue gap measurement. Asituationally aware surgical hub can be configured to infer whether asurgical procedure being performed is a thoracic or an abdominalprocedure, allowing the surgical hub to determine whether the tissueclamped by an end effector of the surgical stapler is lung tissue (for athoracic procedure) or stomach tissue (for an abdominal procedure). Thesurgical hub can then be configured to cause adjustment of thecompression rate and load thresholds of the surgical staplerappropriately for the type of tissue, e.g., by changing or causingchange of at least one variable parameter of an algorithm for thesurgical stapler regarding compression rate and load threshold.

As yet another example, a type of body cavity being operated in duringan insufflation procedure can affect the function of a smoke evacuator.A situationally aware surgical hub can be configured to determinewhether the surgical site is under pressure (by determining that thesurgical procedure is utilizing insufflation) and determine theprocedure type. As a procedure type is generally performed in a specificbody cavity, the surgical hub can be configured to control a motor rateof the smoke evacuator appropriately for the body cavity being operatedin, e.g., by changing or causing change of at least one variableparameter of an algorithm for the smoke evacuator regarding motor rate.Thus, a situationally aware surgical hub may provide a consistent amountof smoke evacuation for both thoracic and abdominal procedures.

As yet another example, a type of procedure being performed can affectthe optimal energy level for an ultrasonic surgical instrument or radiofrequency (RF) electrosurgical instrument to operate at. Arthroscopicprocedures, for example, require higher energy levels because an endeffector of the ultrasonic surgical instrument or RF electrosurgicalinstrument is immersed in fluid. A situationally aware surgical hub canbe configured to determine whether the surgical procedure is anarthroscopic procedure. The surgical hub can be configured to adjust anRF power level or an ultrasonic amplitude of the generator (e.g., adjustenergy level) to compensate for the fluid filled environment, e.g., bychanging or causing change of at least one variable parameter of analgorithm for the instrument and/or a generator regarding energy level.Relatedly, a type of tissue being operated on can affect the optimalenergy level for an ultrasonic surgical instrument or RF electrosurgicalinstrument to operate at. A situationally aware surgical hub can beconfigured to determine what type of surgical procedure is beingperformed and then customize the energy level for the ultrasonicsurgical instrument or RF electrosurgical instrument, respectively,according to the expected tissue profile for the surgical procedure,e.g., by changing or causing change of at least one variable parameterof an algorithm for the instrument and/or a generator regarding energylevel. Furthermore, a situationally aware surgical hub can be configuredto adjust the energy level for the ultrasonic surgical instrument or RFelectrosurgical instrument throughout the course of a surgicalprocedure, rather than just on a procedure-by-procedure basis. Asituationally aware surgical hub can be configured to determine whatstep of the surgical procedure is being performed or will subsequentlybe performed and then update the control algorithm(s) for the generatorand/or ultrasonic surgical instrument or RF electrosurgical instrumentto set the energy level at a value appropriate for the expected tissuetype according to the surgical procedure step.

As another example, a situationally aware surgical hub can be configuredto determine whether the current or subsequent step of a surgicalprocedure requires a different view or degree of magnification on adisplay according to feature(s) at the surgical site that the surgeonand/or other medical practitioner is expected to need to view. Thesurgical hub can be configured to proactively change the displayed view(supplied by, e.g., an imaging device for a visualization system)accordingly so that the display automatically adjusts throughout thesurgical procedure.

As yet another example, a situationally aware surgical hub can beconfigured to determine which step of a surgical procedure is beingperformed or will subsequently be performed and whether particular dataor comparisons between data will be required for that step of thesurgical procedure. The surgical hub can be configured to automaticallycall up data screens based upon the step of the surgical procedure beingperformed, without waiting for the surgeon or other medical practitionerto ask for the particular information.

As another example, a situationally aware surgical hub can be configuredto determine whether a surgeon and/or other medical practitioner ismaking an error or otherwise deviating from an expected course of actionduring the course of a surgical procedure, e.g., as provided in apreoperative surgical plan. For example, the surgical hub can beconfigured to determine a type of surgical procedure being performed,retrieve a corresponding list of steps or order of equipment usage(e.g., from a memory), and then compare the steps being performed or theequipment being used during the course of the surgical procedure to theexpected steps or equipment for the type of surgical procedure that thesurgical hub determined is being performed. The surgical hub can beconfigured to provide an alert (visual, audible, and/or tactile)indicating that an unexpected action is being performed or an unexpecteddevice is being utilized at the particular step in the surgicalprocedure.

In certain instances, operation of a robotic surgical system, such asany of the various robotic surgical systems described herein, can becontrolled by the surgical hub based on its situational awareness and/orfeedback from the components thereof and/or based on information from acloud (e.g., the cloud 713 of FIG. 18 ).

Embodiments of situational awareness systems and using situationalawareness systems during performance of a surgical procedure aredescribed further in previously mentioned U.S. Pat. App. No. 16/729,772entitled “Analyzing Surgical Trends By A Surgical System” filed Dec. 30,2019, U.S. Pat. App. No. 16/729,747 entitled “Dynamic SurgicalVisualization Systems” filed Dec. 30, 2019, U.S. Pat. App. No.16/729,744 entitled “Visualization Systems Using Structured Light” filedDec. 30, 2019, U.S. Pat. App. No. 16/729,778 entitled “System And MethodFor Determining, Adjusting, And Managing Resection Margin About ASubject Tissue” filed Dec. 30, 2019, U.S. Pat. App. No. 16/729,729entitled “Surgical Systems For Proposing And Corroborating Organ PortionRemovals” filed Dec. 30, 2019, U.S. Pat. App. No. 16/729,778 entitled“Surgical System For Overlaying Surgical Instrument Data Onto A VirtualThree Dimensional Construct Of An Organ” filed Dec. 30, 2019, U.S. Pat.App. No. 16/729,751 entitled “Surgical Systems For Generating ThreeDimensional Constructs Of Anatomical Organs And Coupling IdentifiedAnatomical Structures Thereto” filed Dec. 30, 2019, U.S. Pat. App. No.16/729,740 entitled “Surgical Systems Correlating Visualization Data AndPowered Surgical Instrument Data” filed Dec. 30, 2019, U.S. Pat. App.No. 16/729,737 entitled “Adaptive Surgical System Control According ToSurgical Smoke Cloud Characteristics” filed Dec. 30, 2019, U.S. Pat.App. No. 16/729,796 entitled “Adaptive Surgical System Control AccordingTo Surgical Smoke Particulate Characteristics” filed Dec. 30, 2019, U.S.Pat. App. No. 16/729,803 entitled “Adaptive Visualization By A SurgicalSystem” filed Dec. 30, 2019, and U.S. Pat. App. No. 16/729,807 entitled“Method Of Using Imaging Devices In Surgery” filed Dec. 30, 2019.

Surgical Procedures of the Lung

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on a lung. For example, alung resection, e.g., a lobectomy, is a surgical procedure in which allor part, e.g., one or more lobes, of a lung is removed. The purpose ofperforming a lung resection is to treat a damaged or diseased lung as aresult of, for example, lung cancer, emphysema, or bronchiectasis.

During a lung resection, the lung or lungs are first deflated, andthereafter one or more incisions are made on the patient’s side betweenthe patient’s ribs to reach the lungs laparoscopically. Surgicalinstruments, such as graspers and a laparoscope, are inserted throughthe incision. Once the infected or damaged area of the lung isidentified, the area is dissected from the lung and removed from the oneor more incisions. The dissected area and the one or more incisions canbe closed, for example, with a surgical stapler or stitches.

Since the lung is deflated during surgery, the lung, or certain portionsthereof, may need to be mobilized to allow the surgical instruments toreach the surgical site. This mobilization can be carried out bygrasping the outer tissue layer of the lung with graspers and applying aforce to the lung through the graspers. However, the pleura andparenchyma of the lung are very fragile and therefore can be easilyripped or torn under the applied force. Additionally, duringmobilization, the graspers can cut off blood supply to one or more areasof the lung.

Further, a breathing tube is placed into the patient’s airway to alloweach lung to be separately inflated during surgery. Inflation of thelung can cause the lung to move and match pre-operative imaging and/orallow the surgeon to check for leaks at the dissected area(s). However,by inflating the whole lung, working space is lost around the lung dueto the filling of the thoracic cavity. Additionally, inflating a wholelung can take time and does not guarantee easy leak detection ifmultiple portions of the lung are operated on during the surgicalprocedure.

Surgical Procedures of the Colon

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on a colon. For example,surgery is often the main treatment for early-stage colon cancers. Thetype of surgery used depends on the stage (extent) of the cancer, whereit is in the colon, and the goal of the surgery. Some early coloncancers (stage 0 and some early stage I tumors) and most polyps can beremoved during a colonoscopy. However, if the cancer has progressed, alocal excision or colectomy may be required. A colectomy is surgery toremove all or part of the colon. In certain instances, nearby lymphnodes are also removed. If only part of the colon is removed, it’scalled a hemicolectomy, partial colectomy, or segmental resection inwhich the surgeon takes out the diseased part of the colon with a smallsegment of non-diseased colon on either side. Usually, about one-fourthto one-third of the colon is removed, depending on the size and locationof the cancer. Major resections of the colon are illustrated in FIG. 22, in which A-B is a right hemicolectomy, A-C is an extended righthemicolectomy, B-C is a transverse colectomy, C-E is a lefthemicolectomy, D-E is a sigmoid colectomy, D-F is an anterior resection,D-G is a (ultra) low anterior resection, D-H is an abdomino-perinealresection, A-D is a subtotal colectomy, A-E is a total colectomy, andA-H is a total procto-colectomy. Once the resection is complete, theremaining intact sections of colon are then reattached.

A colectomy can be performed through an open colectomy, where a singleincision through the abdominal wall is used to access the colon forseparation and removal of the affected colon tissue, and through alaparoscopic-assisted colectomy. With a laparoscopic-assisted colectomy,the surgery is done through many smaller incisions with surgicalinstruments and a laparoscope passing through the small incisions toremove the entire colon or a part thereof. At the beginning of theprocedure, the abdomen is inflated with gas, e.g., carbon dioxide, toprovide a working space for the surgeon. The laparoscope transmitsimages inside the abdominal cavity, giving the surgeon a magnified viewof the patient’s internal organs on a monitor or other display. Severalother cannulas are inserted to allow the surgeon to work inside andremove part(s) of the colon. Once the diseased parts of the colon areremoved, the remaining ends of the colon are attached to each other,e.g., via staplers or stitches. The entire procedure may be completedthrough the cannulas or by lengthening one of the small cannulaincisions.

During a laparoscopic-assisted colectomy procedure, it is oftendifficult to obtain an adequate operative field. Oftentimes, dissectionsare made deep in the pelvis which makes it difficult to obtain adequatevisualization of the area. As a result, the lower rectum must be liftedand rotated to gain access to the veins and arteries around both sidesof the rectum during mobilization. During manipulation of the lowerrectum, bunching of tissue and/or overstretching of tissue can occur.Additionally, a tumor within the rectum can cause adhesions in thesurrounding pelvis, and as a result, this can require freeing the rectalstump and mobilizing the mesentery and blood supply before transectionand removal of the tumor.

Further, multiple graspers are needed to position the tumor for removalfrom the colon. During dissection of the colon, the tumor should beplaced under tension, which requires grasping and stretching thesurrounding healthy tissue of the colon. However, the manipulating ofthe tissue surrounding the tumor can suffer from reduced blood flow andtrauma due to the graspers placing a high grip force on the tissue.Additionally, during a colectomy, the transverse colon and upperdescending colon may need to be mobilized to allow the healthy, goodremaining colon to be brought down to connect to the rectal stump afterthe section of the colon containing the tumor is transected and removed.

After a colectomy, the remaining healthy portions of the colon must bereattached to one another to create a path for waste to leave the body.However, when using laparoscopic instruments to perform the colectomy,one single entry port may not have a large enough range of motion tomove the one end of the colon to a connecting portion of the colon. Assuch, a second entry port is therefore needed to laparoscopically insertsurgical instruments to help mobilize the colon in order to properlyposition the colon.

Surgical Procedures of the Stomach

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on a stomach. For example,surgery is the most common treatment for stomach cancer. When surgery isrequired for stomach cancer, the goal is to remove the entire tumor aswell as a good margin of healthy stomach tissue around the tumor.Different procedures can be used to remove stomach cancer. The type ofprocedure used depends on what part of the stomach the cancer is locatedand how far it has grown into nearby areas. For example, endoscopicmucosal resection (EMR) and endoscopic submucosal dissection (ESD) areprocedures on the stomach can be used to treat some early-stage cancers.These procedures do not require a cut in the skin, but instead thesurgeon passes an endoscope down the throat and into the stomach of thepatient. Surgical tools (e.g., MEGADYNE™ Tissue Dissector orElectrosurgical Pencils) are then passed through the working channel ofthe endoscope to remove the tumor and some layers of the normal stomachwall below and around it.

Other surgical procedures performed on a stomach include a subtotal(partial) or a total gastrectomy that can be performed as an openprocedure. e.g., surgical instruments are inserted through a largeincision in the skin of the abdomen, or as a laparoscopic procedure,e.g., surgical instruments are inserted into the abdomen through severalsmall cuts. For example, a laparoscopic gastrectomy procedure generallyinvolves insufflation of the abdominal cavity with carbon dioxide gas toa pressure of around 15 millimeters of mercury (mm Hg). The abdominalwall is pierced and a straight tubular cannula or trocar, such as acannula or trocar having a diameter in a range of about 5 mm to about 10mm, is then inserted into the abdominal cavity. A laparoscope connectedto an operating room monitor is used to visualize the operative fieldand is placed through one of the trocar(s). Laparoscopic surgicalinstruments are placed through two or more additional cannulas ortrocars for manipulation by medical practitioner(s), e.g., surgeon andsurgical assistant(s), to remove the desired portion(s) of the stomach.

In certain instances, laparoscopic and endoscopic cooperative surgerycan be used to remove gastric tumors. This cooperative surgery typicallyinvolves introduction of an endoscope, e.g., a gastroscope, andlaparoscopic trocars. A laparoscope and tissue manipulation anddissection surgical instruments are introduced through the trocar. Thetumor location can be identified via the endoscope and a cutting elementthat is inserted into the working channel of the endoscope is then usedfor submucosal resection around the tumor. A laparoscopic dissectionsurgical instrument is then used for seromuscular dissection adjacentthe tumor margins to create an incision through the stomach wall. Thetumor is then pivoted through this incision from the intraluminal space,e.g., inside the stomach, to the extraluminal space, e.g., outside ofthe stomach. A laparoscopic surgical instrument, e.g., an endocutter,can be used to then complete the transection of the tumor from thestomach wall and seal the incision.

Surgical Procedures of the Intestine

Various aspects of the devices, systems, and methods described hereinmay relate to a surgical procedure performed on an intestine. Forexample, a duodenal mucosal resurfacing (DMR) procedure can be performedendoscopically to treat insulin-resistant metabolic diseases such astype 2 diabetes. The DMR procedure can be an effective treatment becauseit affects detection of food. The DMR procedure inhibits duodenumfunction such that food tends to be sensed deeper in the intestine thannormal, e.g., sensed after passage through the duodenum (which is thefirst part of the small intestine). The patient’s body thus senses sugardeeper in the intestine than is typical and thus reacts to the sugarlater than is typical such that glycemic control can be improved. Theirregular function of the duodenum changes the body’s typical responseto the food and, through nervous system and chemical signals, causes thebody to adapt its response to the glucose level to increase insulinlevels.

In the DMR procedure, the duodenal mucosa is lifted, such as withsaline, and then the mucosa is ablated, e.g., using an ablation deviceadvanced into the duodenum through a working channel of an endoscope.Lifting the mucosa before ablation helps protect the duodenum’s outerlayers from being damaged by the ablation. After the mucosa is ablated,the mucosa later regenerates. Examples of ablation devices are NeuWave™ablation probes (available from Ethicon US LLC of Cincinnati, OH).Another example of an ablation device is the Hyblate catheter ablationprobe (available from Hyblate Medical of Misgav, Israel). Anotherexample of an ablation device is the Barxx™ HaloFlex (available fromMedtronic of Minneapolis, MN).

FIG. 22A illustrates one embodiment of a DMR procedure. As shown in FIG.22A, a laparoscope 1400 is positioned external to a duodenum 1402 forexternal visualization of the duodenum 1402. An endoscope 1404 isadvanced transorally through an esophagus 1406, through a stomach 1408,and into the duodenum 1402 for internal visualization of the duodenum1402. An ablation device 1410 is advanced through a working channel ofthe endoscope 1404 to extend distally from the endoscope 1404 into theduodenum 1402. A balloon 1412 of the ablation device 1410 is shownexpanded or inflated in FIG. 22A. The expanded or inflated balloon 1412can help center the ablation device’s electrode so even circumferentialablating can occur before the ablation device 1410 is advanced and/orretracted to repeat ablation. Before the mucosa is ablated using theablation device 1410, the duodenal mucosa is lifted, such as withsaline. In some embodiments in addition to or instead of including theballoon 1412, the ablation device 1410 can be expandable/collapsibleusing an electrode array or basket configured to expand and collapse.

The laparoscope’s external visualization of the duodenum 1402 can allowfor thermal monitoring of the duodenum 1402, which may help ensure thatthe outer layers of the duodenum 1402 are not damaged by the ablation ofthe duodenal mucosa, such as by the duodenum being perforated. Variousembodiments of thermal monitoring are discussed further, for example,below and in U.S. Pat. App No. 17/494,364 entitled “Surgical Methods ForControl Of One Visualization With Another” filed on Oct. 5, 2021. Theendoscope 1404 and/or the ablation device 1410 can include a fiducialmarker thereon that the laparoscope 1400 can be configured to visualizethrough the duodenum’s tissue, e.g., by using invisible light, to helpdetermine where the laparoscope 1400 should externally visualize theduodenum 1402 at a location where ablation occurs. Various embodimentsof fiducial markers are discussed further, for example, in U.S. Pat. AppNo. 17/493,913 entitled “Surgical Methods Using Fiducial IdentificationAnd Tracking” filed on Oct. 5, 2021 and in U.S. Pat. App No. 17/494,364entitled “Surgical Methods For Control Of One Visualization WithAnother” filed on Oct. 5, 2021.

Cooperative Surgical Visualization

Devices, systems, and methods for multi-source imaging provided hereinmay allow for cooperative surgical visualization. In general, incooperative surgical visualization, first and second imaging systemseach gathering images of a surgical site are configured to cooperate toprovide enhanced imaging of a surgical site. The first imaging systemcan be configured similar to the imaging system of FIG. 1 that includesthe imaging device 120, the imaging system 142 of FIG. 4 , the imagingsystem of FIG. 11 , or other imaging system described herein, and thesecond imaging system can be configured similar to the imaging system ofFIG. 1 that includes the imaging device 120, the imaging system 142 ofFIG. 4 , the imaging system of FIG. 11 , or other imaging systemdescribed herein. The cooperative surgical visualization may improvevisualization of patient anatomy at the surgical site and/or improvecontrol of surgical instrument(s) at the surgical site.

Each of the first and second imaging systems is configured to gatherimages of the surgical site from a different point of view by the firstand second imaging systems each viewing the surgical site from adifferent vantage point. The first imaging system can provide a primaryvisualization of the surgical site, while the second imaging system’svisualization of the surgical site can provide a secondary visualizationof the surgical site used to enhance the primary visualization asdisplayed on a display. A single display may thus provide two separatepoints of view of the surgical site, which may conveniently allow amedical practitioner to view only one display instead of multipledisplays. The second imaging system viewing the same surgical site asthe first imaging system but from a different perspective allows for theenhancement since a second view, as gathered by the second imagingsystem, may provide different information than the first imaging system.For example, the first imaging system can be visualizing the surgicalsite from a first side of a tissue wall at the surgical site, and thesecond imaging system can be visualizing the surgical site from asecond, opposite side of the tissue wall. The display can show thesurgical site as visualized by the first imaging system enhanced withimages gathered by the second imaging system that the first imagingsystem cannot gather because the tissue walls blocks the first imagingsystem from such a view. For another example, the first imaging systemcan be visualizing the surgical site from inside a body lumen at thesurgical site, and the second imaging system can be visualizing thesurgical site from outside the body lumen. The display can show thesurgical site as visualized by the first imaging system enhanced withimages gathered by the second imaging system that the first imagingsystem cannot gather because a wall of the body lumen blocks the firstimaging system from such a view. For yet another example, the firstimaging system can be visualizing an organ or other anatomic structureat the surgical site from a first point of view, and the second imagingsystem can be visualizing the organ or other anatomic structure from asecond, different point of view. The display can show the surgical siteas visualized by the first imaging system enhanced with images gatheredby the second imaging system that the first imaging system cannot gatherbecause of the organ or other anatomic structure blocks the firstimaging system from such a view.

Cooperative surgical visualization may allow a critical structure (e.g.,the critical structure 101 of FIG. 1 , the critical structure 401 ofFIG. 15 , the first critical structure 501 a of FIG. 17 , the secondcritical structure 501 b of FIG. 17 , or other critical structuredescribed herein) to be visualized from two different points of view,e.g., from the two different points of view of the first and secondimaging systems, and thus provide a medical practitioner with better,more accurate information about how to avoid the critical structure orhow to access the critical structure as needed in the particularsurgical procedure being performed. In some instances, only one of thefirst and second imaging systems may be able to see a critical structure(e.g., the critical structure 101 of FIG. 1 , the critical structure 401of FIG. 15 , the first critical structure 501 a of FIG. 17 , the secondcritical structure 501 b of FIG. 17 , or other critical structuredescribed herein) from its point of view, in which case the cooperativesurgical visualization may allow the critical structure to be seen whenit would otherwise not be visible if a medical practitioner was onlyviewing images gathered by the other imaging system.

The first and second imaging systems are each configured to be incommunication with a surgical hub (e.g., the surgical hub 706 of FIG. 18, the robotic system surgical hub 822 of FIG. 19 , or other surgical hubdescribed herein) that can be configured to control the positioningand/or orientation of the second imaging system. The surgical hub canthus be configured to receive images gathered by each of the first andsecond imaging systems to facilitate the cooperative surgicalvisualization. In other embodiments, the first and second imagingsystems can be each in communication with a robotic surgical system orother computer system to facilitate the cooperative surgicalvisualization.

In some embodiments, cooperative surgical visualization includes a firstimaging system visualizing a surgical site and coupling a second imagingsystem with the first imaging system, where the second imaging is alsovisualizing the surgical site but from a different point of view thanthe first imaging system. The second imaging system may not have a clearview, or any view, of the critical structure. The first imaging systemcan be used to aid in the positioning and/or orientation of the secondimaging system and thus help adjust the point of view the second imagingsystem has of the surgical site and thereby help improve visualizationof the critical structure. Such adjustment can be particularly usefulduring a tissue interaction activity, such as when the tissue’sconfiguration is being changed during performance of a surgicalprocedure.

For example, in a surgical procedure on a lung such as a lung resection,the lung needs to be collapsed (deflated) for effective performance ofthe surgical procedure. The lung being collapsed is an example of atissue interaction activity. Once the lung is collapsed, the lung is ina different state than shown in any preoperative images and/orintraoperative images of the lung gathered before the lung wascollapsed, such as from a CT or MRI scan. It may therefore be difficultfor a medical practitioner to locate a tumor and/or other criticalstructure(s) with the lung collapsed based on the images in which thelung is not collapsed. Before the lung is deflated, the first and secondimaging systems can each be visualizing the lung, with at least thesecond imaging system visualizing the tumor and/or other criticalstructure(s). Then, during the deflating of the lung, the visualizationprovided by the first imaging system can be used, e.g., by a surgicalhub, a robotic surgical system, or other computer system, to adjust theposition and/or orientation of the second imaging system to help thesecond imaging system maintain and/or provide visualization of the tumorand/or other critical structure(s). The tumor and/or other criticalstructure(s) may thus still be easily located with the lung collapsedbecause at least the second imaging system will be visualizing the tumorand/or other critical structure(s) due to itsrepositioning/reorientation during collapse of the lung.

For another example, a first imaging system, e.g., bronchoscope, can beintroduced into a patient trans-bronchially in a surgical procedure on alung. The first imaging system can thus provide visualization of thelung from a known trans-bronchial position. The visualization providedby the first imaging system from the known trans-bronchial position canbe used, e.g., by a surgical hub, to adjust the position and/ororientation of the second imaging system to help the second imagingsystem maintain and/or provide visualization of the tumor and/or othercritical structure(s).

In some embodiments, cooperative surgical visualization includesproviding shared visualization on a display from a first imaging systemvisualizing a surgical site and a second imaging system also visualizingthe surgical site but from a different point of view than the firstimaging system. For example, some surgical procedures, such as somesurgical procedures of the stomach, some surgical procedures of thelung, and some surgical procedures of the intestine, involvelaparoscopic and endoscopic cooperative surgery in which a laparoscopeand an endoscope are each used to visualize a surgical site. Thelaparoscope can be controlled by a first medical practitioner viewingimages gathered by the laparoscope on a first display, and the endoscopecan be controlled by a second medical practitioner viewing imagesgathered by the endoscope on a second, different display. The medicalpractitioner control can be manual control or control via a roboticsurgical system. Images gathered by each of the first and second imagingsystems, e.g., the laparoscope and the endoscope, can each becommunicated to a surgical hub or other computer system, therebyallowing the surgical hub or other computer system to provide the imagesfrom each of the first and second imaging systems to each of the firstand second displays. Each of the first and second medical practitionersmay therefore be able to more accurately control their respectiveimaging systems since they can see the surgical site from multiplepoints of view on a single display despite each medical practitioneronly controlling an imaging system with one point of view.

The images can be provided to each of the first and second displayswhile each of the first and second imaging systems are moving and thusnot providing images from a static point of view. The first and secondmedical practitioners may thus be able to adjust position and/ororientation of their respective imaging systems based on what the otherimaging system is seeing, e.g., to prevent views that are too similar.Alternatively, the images can be provided to each of the first andsecond displays once at least one of the first and second imagingsystems has stopped moving and is thus providing images from a staticpoint of view. The static point of view may facilitate combined imagingby allowing each of the first and second medical practitioners tocontrol movement and positioning of their respective imaging systemsbased on the view therefrom during setup and a tissue interactionactivity and, thereafter, based on views from both of the first andsecond imaging systems.

Instead of the first and second imaging systems being controlled byfirst and second medical practitioners, the first and second imagingsystems can be controlled by one medical practitioner, either manuallyor via a robotic surgical system. In such instances, the displayassociated with the medical practitioner, e.g., the display that themedical practitioner is viewing whether the display be of a roboticsurgical console or otherwise, can switch back and forth between showingimages gathered by the first imaging system and images gathered by thesecond imaging system while each of the first and second imaging systemsare moving. Then, once at least one of the first and second imagingsystems has stopped moving, the display can provide combined imagingfrom the first and second imaging systems. The static point of view mayfacilitate combined imaging by allowing the medical practitioner tocontrol movement and positioning of the moving one of the imaging systembased on views from both of the first and second imaging systems.

In some embodiments, cooperative surgical visualization includes a firstimaging system visualizing a first anatomic aspect at a surgical siteand a second imaging system visualizing a second, different anatomicaspect at the surgical site where the first imaging system cannotvisualize the second anatomic aspect and the second imaging systemcannot visualize the first anatomic aspect. The first imaging system isconfigured to detect visible light, and the second imaging system isconfigured to detect invisible light such that the first and secondimaging systems are configured to detect different anatomic aspects. Thedifferent anatomic aspects may facilitate identification of a criticalstructure’s location and thereby facilitate positioning of at least oneof the first and second imaging systems and/or facilitate positioning ofat least one instrument at the surgical site. The first imaging systemis configured to gather real time images during a surgical procedure,such as with a scope. The images gathered by the second imaging systemcan be preoperative images or can be real time images gathered duringthe surgical procedure. The first imaging system can thus be controlledbased on the patient’s known anatomy as indicated by images gathered bythe second imaging system as well as on the images gathered by the firstimaging system indicative of its current location and orientation.

For example, a surgical hub, a robotic surgical system, or othercomputer system can be configured to use preoperative and/or real timeimages of a patient, such as CT and/or MRI images, gathered by a secondimaging system in controlling a scope during performance of a surgicalprocedure on the patient. The scope’s movement can thus be controlledbased on the patient’s known anatomy as indicated by preoperative dataas well as on the data gathered by the scope indicative of its currentlocation and orientation. The preoperative images provide anatomicinformation of the patient, for example vasculature and/or tumorinformation. FIG. 23 shows an example of preoperative CT image data of alung of a patient and an example of preoperative MRI image data of thelung of the patient. A lung is shown by way of example. Data gatheredusing invisible light may show another body structure, as appropriatefor the surgical procedure being performed. As a lung is used in thisexample, the scope is a bronchoscope, although other types of scopes canbe used as appropriate for the procedure being performed and the anatomyaccessed. Each of CT image data and MRI image data can be used incontrolling the scope, or only one of CT image data and MRI image datacan be used. Additionally, as mentioned above, other types of invisiblelight image data is possible. Image data from the first and secondimaging systems may also be shown on a display, as discussed herein.

The control of the scope’s movement can include a rate of the scope’sadvancement to the surgical site, a distance of the scope’s advancementto the surgical site, and/or an angle at which the scope is positionedrelative to the surgical site. The preoperative images can help informthe rate of the scope’s advancement to the surgical site since thescope’s current position and the location of the surgical site will eachbe known. The scope can be advanced more quickly the farther away thescope is from the surgical site and then advanced more slowly as thescope gets closer to or is at the surgical site so the scope’s movementcan be more granularly controlled at the surgical site to achieve adesired view and/or avoid or access a critical structure as needed.Similarly, the preoperative images can help inform the distance of thescope’s advancement to the surgical site with the scope being controlledto move greater distances the farther away the scope is from thesurgical site. The preoperative images can help inform the angle atwhich the scope is positioned relative to the surgical site because thepreoperative images can show a critical structure that the scope cannotsee using visible light. The scope can thus be positioned at aneffective orientation relative to the critical structure so that aninstrument can be advanced through the scope to effectively access thecritical structure, such as by passing through healthy tissue to anunderlying tumor that the scope itself cannot visualize (at least notwithout the healthy tissue first being cut to expose the underlyingtumor).

In some embodiments, cooperative surgical visualization includestracking of an anatomic location as it undergoes distortions to predictan internal aspect of the anatomy, e.g., predict a tumor. The anatomiclocation can thus be tracked over two different states, an undistortedstate and a distorted state. The undistorted state can provideinformation useful in determining information, e.g., size, location,shape, and/or orientation, about a tumor or other critical structure inthe distorted state.

As mentioned above, in a surgical procedure on a lung such as a lungresection, the lung needs to be collapsed (deflated) for effectiveperformance of the surgical procedure. However, the collapse can make itdifficult for a medical practitioner to locate a critical structure suchas a tumor based on preoperative and/or intraoperative images in whichthe lung is not collapsed because the lung collapse can cause the tumorto shift in location and/or change size, shape, and/or orientation.Introducing an imaging system such as a flexible scope into the lungwhile the lung is inflated can help a surgical hub, a robotic surgicalsystem, or other computer system more easily coordinate the tumor’slocation as shown via CT imaging with visual and ultrasonic imaging. Thescope can be retracted, such as by being automatically retracted by arobotic surgical system controlling the scope, as the lung is deflatedto prevent the scope from causing damage to tissue adjacent to the scopewith the lung in its deflated state.

FIG. 24 shows one embodiment in which a bronchoscope 1000 has beenadvanced into a lung 1002 and a camera 1004 has been positioned tovisualize the lung 1002. The lung 1002 is collapsed in FIG. 24 . Thebronchoscope 1000 can be introduced via a CT assisted positioning to bedesirably positioned relative to a tumor 1006 at the surgical site orcan use an ultrasound sensor (obscured in FIG. 24 ) at a distal end ofthe bronchoscope 1000 to be desirably positioned relative to the tumor1006. FIG. 24 illustrates with a dotted line 1008 around the tumor 1006ultrasound visualization of the tumor 1006 via the ultrasound sensor.The dotted line 1008 can be provided on a display to inform a medicalpractitioner of the tumor’s detected location and extent (size),information which may not be known without invisible imaging such asultrasound. The dotted line 1008 can be adjusted, such as by a surgicalhub, a robotic surgical system, or other computer system, to be largerthan the tumor 1006 so as to account for a preplanned margin around thetumor 1006 planned to be excised along with the tumor 1006. The dottedline 1008 can be shown on a display to indicate to a medicalpractitioner where the tumor 1006 is located even if the tumor 1006 isnot visible to the naked eye by being hidden under various structure(s).The dotted line 1008 can be three-dimensional so as to provide 3Dviewing of the tumor 1006. Instead of or in addition to ultrasound, oneor more other imaging techniques can be used to facilitate locating thetumor 1006, as discussed herein. The dotted line 1008 can dynamicallychange as the display viewpoint changes and/or as the lung 1002 iscollapsed. The dotted line 1008 is an example only, the tumor’s locationcan be identified in other ways, such as by a particularly colored line,by a line defined by a pattern other than dots, by a translucent shapebeing overlaid on the visualized display at the tumor’s location, or inanother way.

FIG. 25 illustrates one embodiment of automatic retraction of thebronchoscope 1000 during collapse of the lung 1002 with arrows 1010showing a direction of the retraction. FIG. 25 shows the collapsed lung1002 in solid outline as being and shows the inflated lung 1002' indotted outline with arrows 1012 showing the lung 1002 collapse. Thebronchoscope’s imaging can be supplemented by preoperative and/orintraoperative CT scan to image the collapsing lung in real time tofacilitate the retraction of the bronchoscope 1000 so it is straightenedand/or otherwise moved safely, such as to not impact the bronchialbranch junction through which the bronchoscope 1000 will pass. One ormore elements associated with the bronchoscope 1000, such as anovertube, a control sheath, and any instruments that have been advancedthrough a working channel of the bronchoscope 1000 to the surgical site,can not be retracted so as to be left at the surgical site, which mayhelp keep track of the identified area of the tumor 1006 (e.g., byleaving the ultrasound sensor at the surgical site (in embodiments inwhich the ultrasound sensor has been advanced through a working channelof the bronchoscope 1000) and/or by leaving a guide wire-sized catheterat the surgical site providing for a variable wavelength emitting lightor LED visible light at the surgical site) and/or may help preventunintended interaction with tissue as the lung is collapsed.

The automatic retraction of the bronchoscope 1000 can be controlledbased on a plurality of factors. A cumulative or weighted effect of thefactors can be used to determine a rate of retraction of thebronchoscope 1000. The factors can include, for example, a distance fromthe bronchoscope 1000 to the intersecting tissue wall and a rate ofchange. The rate of change can be a rate of change of lung volume (e.g.,CT or ventilator exchange) or a rate of change of the tissue externalsurface (e.g., as indicated by structured light). FIG. 26 shows a graphindicating a rate of retraction of the bronchoscope 1000 in mm/sec (thebronchoscope 1000 is identified as an “endoscope” in the graph) based ona first factor of lung volume reduction (in cc) and a second factor ofbronchoscope 1000 a distance “a” from the bronchoscope 1000 to theintersecting tissue wall (in mm). FIG. 27 illustrates the distance “a.”

A lung can be collapsed without the bronchoscope 1000 having first beenadvanced therein, such that no automatic retraction of a bronchoscope isneeded during lung collapse. Without the bronchoscope 1000 in positionprior to lung collapse, the tumor 1006 may not be visualized using theultrasound sensor associated with the bronchoscope 1000 before lungcollapse. CT imaging and/or structured light imaging can be used tocompare the inflated lung 1002' to the collapsed lung 1002 to track thelocation and size of the tumor 1006 to facilitate advancement of thebronchoscope 1000 to the tumor 1006 after lung collapse. FIG. 28illustrates the inflated lung 1002' and the location and size of thetumor 1006' with the lung 1002' inflated, and illustrates the deflatedlung 1002 and the location and size of the tumor 1006 with the lung 1002deflated. FIG. 28 also illustrates a working space 1014 made availableafter lung collapse for, e.g., tools to be effectively positioned in thepatient.

FIG. 29 shows an enlarged detail of the collapsed-state tumor 1006 andinflated-state tumor 1006', which demonstrates how a tumor can beaffected and change between lung inflation and deflation, thus alsodemonstrating how lung collapse can adversely affect being able tolocate and access a tumor after lung collapse. The illustrated tumor1006, 1006' is a ground-glass opacity nodule (GGN) and is an exampleonly. Other tumors, even other GGN tumors, can have different locations,compositions, and sizes and can accordingly change differently inresponse to lung collapse.

The lung 1002' can be collapsed to control collapse of the tumor 1006'to create a predefined tumor size and shape, e.g., a predefined size andshape of the collapsed-state tumor 1006. The predefined tumor size andshape can correspond to a preferred size and shape for tumor ablation,such as by the collapsed-state tumor 1006 having a size and shape thatcorrelates to microwave antenna performance and/or to a zone of theablation for optimal energy delivery (e.g., an area of ablation that anablation device can create). The predefined tumor size and shape canthus be different based on the particular ablation device being used.The controlled lung collapse may facilitate patient treatment byfacilitating access to the collapsed-state tumor 106 and thus facilitateeffective ablation of the tumor 1006 with the lung 1002 collapsed.Examples of ablation devices are NeuWave™ ablation probes (availablefrom Ethicon US LLC of Cincinnati, OH), which have an ablation zone of 2cm. Another example of an ablation device is the Hyblate catheterablation probe (available from Hyblate Medical of Misgav, Israel).Another example of an ablation device is the Barxx™ HaloFlex (availablefrom Medtronic of Minneapolis, MN). Other examples of ablation devicesare discussed further in, for example, U.S. Pat. App No. 17/494,364entitled “Surgical Methods For Control Of One Visualization WithAnother” filed on Oct. 5, 2021.

An amount of lung collapse during lung deflation can be monitored in X,Y, and Z directions using imaging such as any one or more of CT, x-ray,ultrasound, visualization using the bronchoscope 1000, and MRI. Theinflated size of the lung is known, e.g., as determined via preoperativeimaging, so visualizing the lung during collapse can indicate the lung’spercentage of collapse as the lung moves from inflated toward beingfully deflated. The imaging during lung collapse can also visualize thetumor so as to indicate a size and shape of the tumor with the lung inits current state of deflation. The tumor’s density may also be capturedby the imaging. When the tumor has reached a size and shape that can beeffectively ablated, e.g., given the ablation device’s microwave antennaperformance and/or the ablation device’s ablation zone, the lungcollapse can be stopped. A lung being collapsed for a long period oftime can hinder the lung’s ability to be re-inflated and/or can allowfluid to build up in the lung cavity. Minimizing an amount that the lungis collapsed by monitoring the tumor during lung deflation and stoppingthe lung deflation when the tumor can be effectively accessed andablated may reduce an amount of time that the lung is in a collapsedstate because the lung has not been fully collapsed (in most situationswith most tumors), may help the lung be more quickly re-inflated, and/orbecause the lung is deflated an optimal amount in the first placewithout having to additionally deflate the lung if not initiallydeflated enough for effective tumor ablation. Controlled lung collapseso the lung collapse is stopped when the tumor has a size and shapewithin the ablation zone of the ablation device being used to deliverenergy to the tumor may help prevent more healthy tissue than isnecessary from being ablated because the lung collapse can cease justafter the tumor has a size and shape for effective ablation. Ablating atumor too close to the pleural wall, such as about 1 cm or closer to thepleural wall, could cause damage to the pleural wall, so controlled lungcollapse stopping before full lung collapse may help keep the tumor at asafe distance from the pleural wall during ablation.

FIG. 30 shows a graph indicating ablation zones with respect to apercentage of lung collapse versus tumor size in X, Y, and Z directions.FIG. 31 shows an optimal percentage of lung collapse to ablate a tumorand minimize therapeutic treatment to healthy tissue, as compared to aninflated lung and a fully delated lung. The optimal percentage in thisillustrated embodiment is 60%. As shown in FIG. 31 , the ablation zonein this illustrated embodiment is elliptical, so having information inmultiple directions may help ensure that the tumor at a particularpercentage of lung collapse can be effectively ablated since a center ofthe ablation zone is not a center of the ablation zone’s shape like witha circle or a square. With the lung inflated (0% collapsed) the size andshape of the tumor exceeds the ablation zone. With the lung fullycollapsed (100% collapsed) there is more healthy tissue in the ablationzone than with the lung inflated, and thus that would be ablated werethe tumor ablated with the lung fully collapsed, than with the lungcollapsed at 60%. In other instances, the ablation zone may be morecircular than elliptical or have a differently sized elliptical shape,e.g., because different ablation devices create differently shapedablation zones, in which case a different amount of lung collapse maycorrespond to a size and shape of the tumor for effective ablation.

Different ablation devices can concentrate energy differently, such assome ablation devices concentrating energy pointing proximally from atip of the ablation device and others concentrating energy pointingdistally from a tip of the ablation device. Using multiple types ofimaging during lung collapse may allow for images of the tumor to begathered that will be helpful in determining where to position theablation device relative to the tumor to help ensure full ablation ofthe tumor while minimizing damage to healthy, non-targeted tissue.

Controlled lung collapse can be performed in a variety of ways, such asby a combination of extracting air from the airway and controlledsealing and venting by the bronchoscope 1000 if advanced into the lung1002 before lung collapse. This technique allows portions of thebronchus airways to be sealed and allow other portions of the bronchusairways to be selectively collapsed, thereby causing air removal andtissue strain from adjacent areas to compact the tumor 1006 into thepredefined size and shape. Which airways to seal, where to positionballoons used to create the seal (in embodiments in which sealing isperformed using inflatable balloons), which airways to evacuate, and howmuch air to evacuate from each of the evacuated airways can bepre-operatively determined by, for example, performing a surgicalsimulation via a surgical hub that is later used operatively to sealairways as sealed in the surgical simulation saved at the surgical huband to evacuate airways as evacuated in the surgical simulation saved atthe surgical hub. In embodiments in which the bronchoscope 1000 was notadvanced into the lung 1002 before lung collapse, controlled lungcollapse can be performed, for example, by extracting air from theairway. Which airways to evacuate and how much air to evacuate from eachof the evacuated airways can be pre-operatively determined by, forexample, performing a surgical simulation via a surgical hub that islater used operatively to evacuate airways as evacuated in the surgicalsimulation saved at the surgical hub.

As mentioned above, the tumor 1006 can be visualized. In some instances,the tumor 1006 cannot be visualized, such as if an imaging device thatcan visualize using invisible light is not being used and tissueoverlies the tumor 1006 so as to prevent visualization of the tumor 1006using visible light. In such instances, the overlying tissue can beimaged and tracked similar to that discussed above regarding the tumor1006 so that the size and shape of the tumor 1006 known to be under thetissue, e.g., because of preoperative imaging, can be predicted based onthe changing size and shape of the overlying tissue during lungcollapse. A dotted line similar to the dotted line 1008 (or indicatorother than a dotted line, as discussed above) can be provided on adisplay to indicate that predicted size and shape of the tumor 1006based on the overlying tissue. The predicted size and shape of the tumor1006 can be used similar to that discussed above in determining how muchto collapse the lung 1002 for effective ablation of the tumor 1006.

A structured light projection, as discussed herein, can be used to imageand track the overlying tissue. For example, FIG. 32 illustrates oneembodiment of using a structured light projection of a laser arraysupplemented by tissue surface marks to map external surface deformationof a lung to provide information about a tumor in the lung. Thedifferences of the two arrays over time as the lung is deformed allowsthe surgical hub, a robotic surgical system, or other computer system tomeasure geometry and tissue strain. FIG. 32 shows the lung 1020 in aninflated state with a corresponding tissue surface geometry 1022, showsthe lung 1020' in a partially collapsed state with a correspondingtissue surface geometry 1022', and shows the lung 1020" in a fullycollapsed state with a corresponding tissue surface geometry 1022". FIG.33 shows physical organ tag markers 1024, which can be ink marks orother tag markers, and projected structured light 1026 corresponding tothe inflated state (lung 1020 and tissue surface geometry 1022). FIG. 34shows the physical organ tag markers 1024 and the projected structuredlight 1026 corresponding to the fully collapsed state (lung 1020" andtissue surface geometry 1022"). With respect to the projected structuredlight 1026, the physical organ tag markers 1024 have become distortedmoving from the inflated state to the partially collapsed state, andfurther distorted moving from the partially collapsed state to the fullycollapsed state. The distortion of the physical organ tag markers 1024with respect to the projected structured light 1026 measures strain ofthe lung. The measured strain can then be used to estimate the size andshape of the tumor underlying the strained tissue. In this way, thetumor’s size and shape can be predicted.

FIG. 35 illustrates one embodiment of a method 1030 of predicting tumorsize using measured tissue strain. The method 1030 is described withrespect to a lung illustrated in FIG. 36 .

In the method 1030, the lung 1056 is imaged 1032 in the inflated state.CT imaging is used in this example but other imaging is possible. Theimaging of the inflated lung 1056 indicates a state of the tissuewithout strain. The lung 1056' is then partially collapsed 1036, to an80% partially collapsed state in this illustrated embodiment.

A surgical device 1050 (a scope in this illustrated embodiment) isintroduced 1036 to the surgical site and positioned relative to thepartially collapsed lung 1056'. An imaging device 1052 is alsointroduced to the surgical site, either before or after the surgicaldevice 1050, and positioned relative to the partially collapsed lung1056'. The surgical device 1050 is similar to the surgical device 102 ofFIG. 1 , the surgical device 202 of FIG. 8 , or other surgical devicedescribed herein, and the imaging device 1052 is similar to the imagingdevice 120 of FIG. 1 , the imaging device 220 of FIG. 8 , or otherimaging device described herein. The surgical device 1050 provides astructured light pattern 1054, as described herein, on the partiallycollapsed lung 1056'. The structured light pattern 1054 is a dot patternin this illustrated embodiment. The imaging device 1052 detects thestructured light pattern 1054, as described herein.

Physical organ tag markers 1058 are on the surface of the lung. Themarkers 1058 are tattooed ink marks in this illustrated embodiment butother markers are possible. The markers 1058 have been applied with thelung 1056 in the inflated state.

With the lung 1056' in the partially collapsed state, a tissue surfacedeformation map is created 1038 based on the structured light pattern1054, as discussed herein. The tissue surface deformation map iscompared 1040 to the inflated lung image to generate 1042 a tissuestrain (or stress) collapsed 3D model of a tumor underlying the lungtissue surface. The generated 1042 3D model can be shown on a display,such as with a dotted line, translucent shape, etc. as described herein,to show the estimated tumor size and shape. FIG. 37 and FIG. 38illustrate the 80% partially collapsed lung 1056' and the estimatedsub-surface geometry of the tumor 1060' with the lung 1056' 80%collapsed. FIG. 38 also shows the estimated sub-surface geometry of thetumor 1060 compared to an ablation zone 1062 of an ablation device to beused to ablate the tumor. The ablation zone 1062 is elliptical in thisillustrated embodiment. As the lung continues to be collapsed, a tissuesurface deformation map is created 1038 based on the structured lightpattern 1054 and compared 1040 to the inflated lung image to generate1042 a tissue strain (or stress) collapsed 3D model of the tumorcorresponding to that state of lung collapse. FIG. 38 shows theestimated sub-surface geometry of the tumor 1060 compared to theablation zone 1062 at 70% lung collapse, 60% lung collapse, 50% lungcollapse, and 40% lung collapse. The estimated tumor geometry can bedisplayed at each of these lung collapse states as well as lung statestherebetween. The tumor’s size and shape can thus be visualized in realtime during performance of the surgical procedure. The tumor 1060 firstfits within the ablation zone 1062 at 60% lung collapse, but the lungcontinues being collapsed to 40% lung collapse in this illustratedembodiment to help ensure full ablation of the tumor. FIG. 38 alsoillustrates the deformation of the tissue at each of the 80%, 70%, 60%,50%, and 40% lung collapse states. FIG. 36 shows the inflated lung 1056,the 80% collapsed lung 1056', the 40% collapsed lung 1056" (also shownin FIG. 37 ), and 0% (fully deflated) lung 1056"'. The lung is not fullydeflated in this method 1030 but a lung can be fully deflated, orpartially deflated at less than 40%, in other embodiments to achieve adesired tumor size and shape.

With the lung 1056" at 40% collapse, which corresponds to the estimatedtumor size and shape being at the predefined size and shapecorresponding to a preferred size and shape for tumor ablation, the lung1056" is again imaged 1044. As mentioned above, CT imaging is used inthis example. The lung 1056" being imaged 1044 allows the actual sizeand shape of the tumor 1060 to be determined. The lung can then beinflated or deflated further as needed to achieve the predefined sizeand shape should the actual tumor 1060 size and shape not sufficientlycorrespond to the estimated tumor 1060 size and shape.

With the lung deflated as desired, the tumor 1060 is ablated. The effectof the ablation can be monitored, such as by using multi-spectralimaging or using electromagnetic or RF monitoring, to determine anactual ablation zone. The actual ablation zone and the overlaid ablationzone 1062 can each be used to enable more precise treatmenttriangulation and feedback control to the ablation system that includesthe ablation device.

In some embodiments, a camera can provide an intraluminal view, such asby a bronchoscope being introduced into the lung in the inflated state.As the lung is collapsed, e.g., as the lung is deflated to the 40%collapsed state, the intraluminal imaging can be used in combinationwith the structured light imaging to determine a shape, thickness, andconfiguration of a tissue wall separating the two visualization systems.Monitoring the shape, thickness, and configuration of the tissue wall isanother way that the tissue distortion or tissue strain can bedetermined. Various embodiments of tissue monitoring are discussedfurther, for example, in U.S. Pat. App No. 17/494,364 entitled “SurgicalMethods For Control Of One Visualization With Another” filed on Oct. 5,2021.

3D Surgical Visualization

Surface irregularities (e.g., deformations and/or discontinuities) ontissue can be difficult to capture and portray in a visualizationsystem. Additionally, tissue often moves and/or changes during asurgical procedure. In other words, tissue is dynamic. For example, thetissue may be distorted, stressed, or become otherwise deformed by asurgical fastening operation. The tissue may also be transected and/orcertain portions and/or layers of tissue may be removed. Underlyingtissue and/or structures may become exposed during the surgicalprocedure. As the tissue moves, embedded structures underlying thevisible tissue and/or hidden tissue margins within an anatomicalstructure may also move. For example, a resection margin may beconcentrically positioned around a tumor prior to tissue deformation;however, as the anatomical structure is deformed during a surgicalprocedure, the resection margin may also become deformed. In certaininstances, adjacent portions of tissue can shift, including thoseportions with previously-identified physical characteristics orproperties. Generating 3D digital representations or models of thetissue as it is deformed, transected, moved, or otherwise changed duringa surgical procedure presents various challenges; however, such dynamicvisualization imaging may be helpful to a clinician in certaininstances.

In various instances, an adaptive visualization system can providevisualization of motion at the surgical site. Structured light can beused with stereoscopic imaging sensors and multi-source coherent lightto map light pattern distortions from one time frame to another timeframe. The mapping of light pattern distortions across frames can beused to visualize and analyze anatomic distortion. Moreover, as the 3Ddigital representations or models are deformed, any superimposed 3Dimaging, such as embedded structures, tissue irregularities, and/orhidden boundaries and/or tissue margins, can be proportionately deformedwith the 3D model. In such instances, the visualization system canconvey movement of the superimposed 3D imaging to a medical practitioneras the tissue is manipulated, e.g., dissected and/or retracted.

An adaptive visualization system can obtain baseline visualization databased on situational awareness (e.g., input from a situational awarenesssystem). For example, a baseline visualization of an anatomicalstructure and/or surgical site can be obtained before initiation of asurgical procedure, such as before the manipulation and dissection oftissue at the surgical site. The baseline visualization image of theanatomical geometry can include a visualization of the surface of theanatomical structure and its boundaries. Such a baseline visualizationimage can be used to preserve overall orientation of the surgical siteand anatomic structure even as local regions within the anatomicstructure are progressively disrupted, altered, or otherwise manipulatedduring the surgical procedure. Maintaining the baseline visualizationimage can allow the disrupted regions to be ignored when mapping otherimaging irregularities. For example, when mapping or overlayingstructures and/or features obtained by other imaging sources, thebaseline visualization image can be used and the distorted regionsignored to appropriately position the additional structures and/orfeatures in the updated visualization image.

Various systems and methods of generating 3D digital representations ormodels of tissue and conveying the 3D models to medical practitionersare further described in, for example, previously mentioned U.S. Pat.App. No. 16/729,747 entitled “Dynamic Surgical Visualization Systems”filed Dec. 30, 2019.

Devices, systems, and methods for multi-source imaging provided hereinmay allow for enhanced 3D surgical visualization. In general, thedevices, systems, and methods can allow identification and accentuationof anatomical gateways by differentiating tissue planes, localizedanomalies, and tissue configuration or composition to enhance a 3D modelbeing used for visualization. In an exemplary embodiment, multiplevisualization views can be used to identify, locate, and defineboundaries of connective soft tissue planes. The defined planes can berelated to a structure and role of the tissue. Examples of the structureand role include tissue composition, tumor location, tumor marginidentification, adhesions, vascularization, and tissue fragility.Irregularities can be localized and overlaid onto the defined tissueplanes on a display. Properties of a displayed tissue plane may bevisible on the display, such as by a medical practitioner highlightingor otherwise selecting the tissue plane. Examples of the propertiesinclude tissue type, collagen composition, organized versus remodeleddisorganized fiber orientation, tissue viability, and tissue health.

In some embodiments, enhanced 3D surgical visualization includesaugmenting cooperative imaging resulting from multiple visualizationsources, e.g., from a plurality of imaging devices, to form a compositemap. The composite map can identify localized aspects, markers, orlandmarks. These enhanced components can be coupled to a surgicalprocedure plan or global anatomy to identify and highlight aspects ofthe tissue that correspond to steps in the surgical procedure plan anddirections improving visualization during performance of the steps.

A CT image of a patient can show a tissue plane between tissue segments.The CT image can be used to locate or identify the tissue plane anddisplay the tissue plane relative to another image of the patientgathered by an imaging device, such as by overlaying a representation ofthe tissue plane on the image gathered by the imaging device and shownon the display. By way of example, FIG. 39 shows a CT image of a lungwith two arrows pointing to an intersegmental plane between segments S₁,S₂, and S₃. Other white lines and areas in the CT image are vessels andairways. FIG. 40 shows another CT image of the lung with one arrowpointing to an intersegmental plane between segments S₁ and S₂. Otherwhite lines and areas in the CT image are vessels and airways.

Multiple sources of visualization can be overlaid to form a singlevisualization on a display. The display can also show a secondary globalview to coordinate imaging views and reduce confusion or loss oforientation. In an exemplary embodiment, the overlaid information caninclude identified tissue plane(s) overlaid on a primary visualizationprovided by a rigid scope. The single visualization can thus definetissue plane location and orientation relative to the primary view todirect tissue plane dissection and separation being executed by a user,such as by the user controlling one or more surgical instrumentsmanually or using a robotic surgical system.

FIG. 41 illustrates one embodiment of a display showing a thoracoscopicview 1100 provided by a rigid scope and showing a tissue plane 1102. Aflexible scope 1106 providing a flexible scope view 1108 is visible bythe rigid scope and is thus shown in the thoracoscopic view 1100. Therigid scope view 1100 also shows first and second surgical instruments1110, 1112 at the surgical site. The surgical instruments 1110, 1112 areeach dissectors configured to dissect tissue, e.g., to separate tissueat the tissue plane 1102, for removal of a specimen 1104, such as atumor, that is also visible in the rigid scope’s view 1100. The displayalso shows a ghosted thoracoscopic view 1114 and a 3D model 1116 showingthe thoracoscopic visualization plane 1118 and the flexible scopevisualization plane 1120.

The display also shows the flexible scope view 1108, which is the viewseen by the flexible scope 1106. A medical practitioner can thus see ona single display what each of the rigid and flexible scopes arevisualizing. The view 1100 provided by the rigid scope is the primaryview on the display, as indicated by its size being larger than theflexible scope view 1108. The medical practitioner can switch betweenwhich view is the primary view, and thus which is the larger view on thedisplay.

FIG. 42 illustrates another embodiment in which a CT image 1122 isenhanced with an overlay 1124 outlining an identified criticalstructure, which is a tumor in this example. Images taken over time canbe used to develop the overlay 1124, as discussed herein. The overlay1124 allows airways to be distinguished from the critical structure. TheCT image 1122 shows a lung, but CT images of other anatomic structurescan be similarly overlaid. Also images other CT images, such asultrasound images or MRI images, could similarly have an overlay.

In some embodiments, enhanced 3D surgical visualization includesadaptive differentiation and mapping convergence of a first imagingsystem via a second imaging system. In other words, enhanced mapping andguidance can be provided using visualizations of a surgical site fromdifferent imaging sources. Images gathered by the first imaging systemcan be enhanced with images from the second imaging system such that thefirst imaging system is the primary system and the second imaging systemis the secondary system. The images gathered by the first imaging systemcan be enhanced by adding information obscured from the first imagingsystem but visible by the second imaging system and/or bydifferentiating aspects of the primary imaging for secondary usage.

For example, in a surgical procedure on a lung, the adaptivedifferentiation and mapping convergence can include progressive flexiblescope imaging of the lung’s airways. A front view of lung airways isshown in FIG. 43 . FIG. 44 and FIG. 45 show one example of a path ofadvancement for a flexible scope 1130 in a lung. As the flexible scope1130 is introduced into the lung, progressive imaging of the lung’sinternal architecture linked to the scope’s tracked position andorientation can be used, e.g., by a surgical hub or other computersystem, to enhance, and confirm or adjust, a CT mapping of the airwayroutes from a moving or indexing starting point to an intendeddestination, such as a tumor in the lung. In this way, as the flexiblescope 1130 advances in the lung, a display can show the scope’sprogressive advancement along the path of advancement and map (which mayinclude altering) any changes in the CT scanned geometry as the flexiblescope 1130 advances. The mapping of local adjacent anatomic structuresas the flexible scope 1130 is advanced into position may improve detailalong the path of advancement, may provide another data point toconfirmation of the primary global CT mapping so as to allow more pointsof fiducial comparison, and/or may allow the medical practitionercontrolling the flexible scope 1130 (and/or other medical practitioner)to define desired local nodal dissection (if any) around a tumor marginor within a zone of confirmation that would allow stoppage during thetumor extraction procedure to dissect out the desired nodes since theirlocations have already been identified and recorded. The progressiveadvancement can be enhanced by a Lidar array. The Lidar array allowsmore precise distance measurement to ultrasound surfaces and thus to theunderlying structure.

FIG. 46 illustrates one embodiment of a display of nodes 1132 that maybe defined and dissected as desired. Six nodes are shown, but adifferent number may be present. FIG. 46 also shows the flexible scope1130, as a radial endobronchial ultrasound (R-EBUS) probe, in an airway1134 en route to its intended destination of a tumor 1136. Progressiveradial ultrasound achieved using the R-EBUS probe 1130 can be used tofacilitate the node 1132 identification and recording. The R-EBUSprobe’s array can be positioned in a radially spinning connection thatallows for circumferential scanning of the anatomy below the locallypresented bronchial surface of the airway 1134 in which the scope 1130is positioned, thereby enabling the scope 1130 to detect nodes, tumors,and irregularities in the anatomy as the scope 1130 is advanced in thelung.

Procedures in other anatomic locations may be similarly mapped andnodes, tumors, and irregularities in the anatomy similarly detected.

For another example, a secondary imaging system using wavelengthsoutside the visible light range can be used to identify and/or improvevisualization features that are obscured from a primary imaging systemusing visible light and/or to augment the primary imaging system’sinformation regarding those features. In an exemplary embodiment, asecondary imaging system uses multi-spectral imaging using non-visiblelight, such as light in infrared (IR) and ultraviolet (UV) spectrums,during a surgical procedure to identify features that are obscured to aprimary imaging system using visible light for visualization. Light inthe ultraviolet spectrum penetrates tissue poorly but can be helpful inidentifying superficial crevices, shapes, and other features.Ultraviolet imaging can be performed, for example, using filters on aCMOS array with a UV source coupled to a primary light source or, foranother example, using a multi-laser 480 Hz UV, R, G, B, IR imagingsystem where each spectrum is pulsed separately to create a composite 60Hz image for display. Light in the infrared spectrum penetrates tissuewell and can thus be helpful in identifying structure covered bysuperficial connective tissue, fat, or other thin tissues that preventvisible light imaging from seeing the underlying structure. Examples ofmulti-spectrum imaging, such as those that can be used to calculatesurface refractivity of tissue to determine the tissue’s composition andirregularities, are further described, for example, in U.S. Pat. Pub.No. 2019/0200905 entitled “Characterization Of Tissue IrregularitiesThrough The Use Of Mono-Chromatic Light Refractivity” published Jul. 4,2019, which is hereby incorporated by reference in its entirety.Examples of 3D imaging using structured light, such as the coupling of3D generated constructs to key anatomic structures to enable thevisualization of tissue parameters, procedure layout plan, predictivedistortion of superimposed images, and instrument characterization, arefurther described, for example, in previously mentioned U.S. Pat. Pub.No. 2021/0196385 entitled “Surgical Systems For Generating ThreeDimensional Constructs Of Anatomical Organs And Coupling IdentifiedAnatomical Structures Thereto” published Jul. 1, 2021.

The features may be obscured (fully or partially) to visible lightimaging due to being covered by fluid, being covered by tissue, having awashed out contrast, having a similar color to adjacent structure(s),and/or having a similar reflectivity to adjacent structure(s). Thesecondary imaging may allow for visualization underneath obscuring fluidand/or tissue that visible light visualization cannot provide. Thesecondary imaging improves contrast of the features and may thus helpidentify features having a washed out contrast, having a similar colorto adjacent structure(s), and/or having a similar reflectivity toadjacent structure(s). A washed out contrast may exist because a visuallight imaging system is typically used with as high a brightness level(brightness intensity) as is possible or tolerable to improvevisualization of the surgical site. However, the high brightness leveltends to wash out the contrast of instruments and tissues, particularlyany tissues directly outside of the focused-on site.

The secondary information gathered by the secondary imaging system canbe provided via overlay on a display showing the primary informationgathered by the primary imaging system (or showing a 3D model generatedusing the primary images), which may allow a medical practitioner tolook at one display for complete visualization information and/or mayhelp the medical practitioner make better decisions on the fly regardingnavigation, dissection, and/or other procedure steps than would bepossible if only the primary imaging was available without theenhancement provided by the secondary imaging. The non-visible light canbe projected at a different intensity than the visible light and/or canbe varied over its exposure to improve resolution of identifiedcontrasts, which may allow a surgical hub, a robotic surgical system, orother computer system (e.g., a controller thereof) to look for contrastseparately from the visible light imaging and then overlay the visibleand non-visible light imaging. If the surgical hub, robotic surgicalsystem, or other computer system detects a contrast that is irregular orinconsistent, the surgical hub, robotic surgical system, or othercomputer system can automatically adjust field-programmable gate array(FPGA) isolation of the contrast to enhance it or can automaticallyadjust the intensity while using the irregular or inconsistent region asa focal guide to adjust contrasting to the portions of the image thatare most difficult to contrast.

FIG. 47 illustrates one embodiment of a method 1140 of using a primaryimaging system utilizing visible light and a secondary imaging systemutilizing wavelengths outside the visible light range to identify and/orimprove visualization features that are obscured from the primaryimaging system. In the method 1140, a surgical hub, a robotic surgicalsystem, or other computer system (e.g., a controller thereof) determines1142 that an image of a structure gathered by an imaging device (a scopein this example) of the primary imaging system are blurry or distorted.The surgical hub, robotic surgical system, or other computer systemcalculates 1144 a distance between the scope and the structure. Based onthe calculated distance, the surgical hub, robotic surgical system, orother computer system determines 1146 whether the scope is within anoptimal working range for visualization. If the calculated distance isabove a predetermined maximum threshold distance, the scope can beconsidered to be outside the optimal working range by being too far awayfrom the structure for optimal non-blurry, non-distorted imaging. If thecalculated distance is below a predetermined threshold minimum distance,the scope can be considered to be outside the optimal working range bybeing too close to the structure for optimal non-blurry, non-distortedimaging.

If the scope is determined 1146 to not be in the optimal working range,the surgical hub, robotic surgical system, or other computer systemcauses 1148 the Z-axis of the scope to move in or out until the distanceis within the optimal working range. The surgical hub, robotic surgicalsystem, or other computer system then determines 1150 whether an imageof the structure gathered by the imaging device, after the causing 1148,is clear. If the image is determined 1150 to be clear, then automaticadjustment of the scope stops 1152 and the surgical procedure continues(invisible to the user). If the image is determined 1150 to not beclear, the surgical hub, robotic surgical system, or other computersystem adjusts 1154 a light source of the primary imaging system andsweeps the Z-axis of the scope in and out within the optimal workingrange, e.g., to the predetermined maximum threshold distance andpredetermined minimum threshold distance. Then, the surgical hub,robotic surgical system, or other computer system then determines 1156whether an image of the structure gathered by the imaging device, afterthe adjustment 1154, is clear. If the image is determined 1156 to beclear, then automatic adjustment of the scope stops 1152 and thesurgical procedure continues (invisible to the user). If the image isdetermined 1156 to not be clear, the surgical hub, robotic surgicalsystem, or other computer system adjusts 1158 a light source of thesecondary imaging system and sweeps the Z-axis of the scope in and outwithin the optimal working range. Then, the surgical hub, roboticsurgical system, or other computer system determines 1160 whether animage of the structure gathered by the imaging device, after theadjustment 1158, is clear. If the image is determined 1160 to be clear,then automatic adjustment of the scope stops 1152 and the surgicalprocedure continues (invisible to the user). If the image is determined1160 to not be clear, the surgical hub, robotic surgical system, orother computer system calculates 1162 best image settings from the firstadjustment 1154 and the second adjustment 1158. Then, the surgical hub,robotic surgical system, or other computer system determines 1164whether an image of the structure gathered by the imaging device, withthe calculated 1162 settings set, is clear. If the image is determined1164 to be clear, then automatic adjustment of the scope stops 1152 andthe surgical procedure continues (invisible to the user). If the imageis determined 1164 to not be clear, the blurred or distorted image islikely due to fluid or other matter obstructing the scope’s lens. Thus,the surgical hub, robotic surgical system, or other computer systemdetermines 1166 whether scope cleaning technology is installed on thescope. If scope cleaning technology is determined 1166 to not beinstalled, the surgical hub, robotic surgical system, or other computersystem causes 1168 a user notification, e.g., a visual alert on adisplay, an audible announcement, etc., to be provided indicating thatthe scope must be removed and cleaned or be replaced with a differentscope. In such an instance, the blurred or distorted image is likely dueto fluid or other matter obstructing the scope’s lens, and no scopecleaning technology is available on the scope to automatically clean thescope at the scope’s current location in the patient. If scope cleaningtechnology is determined 1166 to be installed, the surgical hub, roboticsurgical system, or other computer system activates 1170 the scopecleaning technology to clean the scope, such as by causing the scope’slens to be sprayed with cleaning fluid. After the activation 1170 of thescope cleaning technology, the first adjustment 1154 occurs again andthe method 1140 continues as discussed above. If the surgical hub,robotic surgical system, or other computer system determines 1164 asecond time that the image is not clear, the method 1140 can continue asdiscussed above or can instead proceed directly to causing 1168 a usernotification since the previous cleaning was unsuccessful.

If the scope is determined 1146 to be in the optimal working range, thesurgical hub, robotic surgical system, or other computer system adjusts1154 the light source of the primary imaging system and the method 1140continues as discussed above. If the surgical hub, robotic surgicalsystem, or other computer system determines 1164 a second time that theimage is not clear, the method 1140 can continue as discussed above orcan instead proceed directly to causing 1168 a user notification sincethe previous cleaning was unsuccessful.

For yet another example, local imaging, such as multi-spectral light orultrasound, can be used to adjust 3D constructs, mapping, or CT imagingbased on identified deviations from a preoperative surgical plan or apreoperative scan to more up-to-date, higher precision, and/or improvedimaging from a local imaging system. The local imaging can be used toadjust global imaging (e.g., a preoperative full body CT planned scan)or focused cone-beam CT intraoperative imaging based on changes,deviations, and/or irregularities identified locally using the localimaging. Examples of using an independent color cascade of illuminationsources including visible light and light outside of the visible rangeto image one or more tissues within a surgical site at different timesand at different depths, and using sequential laser light from differingsources to IR-R-B-G-UV coloration and determining properties of backscattered light and Doppler effect to track moving particles, arefurther described, for example, in U.S. Pat. Pub. No. 2019/0206050entitled “Use Of Laser Light And Red-Green-Blue Coloration To DetermineProperties Of Back Scattered Light” published Jul. 4, 2019, which ishereby incorporated by reference in its entirety.

In an exemplary embodiment, a contrasting agent can be introduced into apatient’s physiologic system to highlight, fluoresce, or contrastanatomic structures intraoperatively. Contrast-enhanced ultrasound withmicrobubbles can be used, such as for blood perfusion in organs,thrombosis (such as in myocardial infraction), abnormalities in theheart, liver masses, kidney masses, inflammatory activity ininflammatory bowel disease, and a chemotherapy treatment response.Microbubble contrast materials are tiny bubbles of an injectable gasheld in a supporting shell. The microbubble shells dissolve in thepatient, usually within a range of about 10 minutes to about 15 minutes,and the gas is removed from the body through exhalation. As anotherexample, indocyanine green (ICG) IT fluorescing can be used, such as forfluorescing blood vessels. As yet another example, barium sulfide andiodine, such as iodine-containing contrast medium (ICCM) can be used,such as for CT blood vessel imaging. As still another example,gadolinium can be used, such as for MRI imaging.

FIG. 48 illustrates one embodiment of a display showing an on-the-flyadaptation of vascular CT with real-time local scanning. The displayincludes a preoperative CT scan 1180 showing anticipated vascularlocation. The display also includes an adaptive real time view 1182 thatincludes real-time local scanning. The adaptive real time view 1182includes ICG real-time vascular imaging 1184 indicating real-timevascular location overlaid with the relevant area 1186 (shown via afirst color dotted line) of the preoperative CT scan 1180 and overlaidwith a real time realigned vascular imaging 1188 (shown via a secondcolor dotted line) based on the local imaging 1184 and the preoperativeCT scan 1180. The overlays are shown with differently colored dottedlines in this illustrated embodiment but can be indicated in other ways,such as by differently styled lines (dotted versus dashed, etc.) of asame color, differently styled lines of different colors, lines ofdifferent brightness, etc.

Controlling Cooperative Surgical Imaging Interactions

Devices, systems, and methods for multi-source imaging provided hereinmay allow for controlling cooperative surgical imaging interactions. Ingeneral, identifying, determining a location of, and determining anorientation of a first and second imaging systems relative to oneanother can allow for controlling cooperative surgical imaginginteractions. In an exemplary embodiment, one of the first and secondimaging systems can be identified and tracked by the other one of theimaging systems with the first and second imaging systems beingpositioned on opposite sides of a tissue wall such that the tissue wallprovides an obstruction preventing the first and second imaging systemsfrom seeing one another.

For example, a first imaging system located on a first side of a tissuewall can be configured to use magnetic sensing to detect a location of asecond imaging system located on a second, opposite side of the tissuewall. FIG. 49 illustrates one embodiment of a first imaging system,including a flexible scope 1200, located on a first side of a tissuewall 1202 configured to use magnetic sensing to detect a location of asecond imaging system, including a rigid scope 1204, located on asecond, opposite side of the tissue wall 1202. The flexible scope 1200includes a first magnetic fiducial marker 1206, and the rigid scope 1204includes a second magnetic fiducial marker 1208. FIG. 49 illustrates alung, but imaging systems can be located on opposite sides of a tissuewall of another anatomic structure. Various embodiments of magneticfiducial markers and using magnetic fiducial markers in detectinglocation are discussed further, for example, in U.S. Pat. App No.17/494,364 entitled “Surgical Methods For Control Of One VisualizationWith Another” filed on Oct. 5, 2021.

For another example, a first imaging system located on a first side of atissue wall can be configured to use common anatomic landmarks to detecta location and orientation of a second imaging system located on asecond, opposite side of the tissue wall. FIG. 50 illustrates oneembodiment of a first imaging system, including a flexible scope 1210,located on a first side of a tissue wall 1212 and a rigid scope 1214,located on a second, opposite side of the tissue wall 1212.

For yet another example, a structured light scan can be used to create a3D map. Electromagnetic tracking of a distal end of a first scope, e.g.,using a fiducial marker such as the distal fiducial marker 1206 or thedistal fiducial marker 1208, provides 3D registration of the map. Aperimeter can then be created around a tumor or other critical structureusing manual line or guides by confocal laser endomicroscopy to providereal time histology guidance. A line as registered in space can becommunicated to a second scope located on an opposite side of a tissuewall as the first scope for pinch assist on the second scope’s side ofthe tissue wall for submucosal resection or geofencing for full wallresection.

For still another example, a surgical instrument can be tracked by animaging system using a fiducial marker on the surgical instrument. Anynumber of surgical instruments can be tracked in this way duringperformance of a surgical procedure. FIG. 51 illustrates one embodimentof a surgical instrument 1220 that includes a fiducial marker 1222 a,1222 b, 1222 c. Three fiducial marker 1222 a, 1222 b, 1222 c are used inthis illustrated embodiment, but another number is possible. FIG. 51shows a camera 1224 and the surgical instrument 1220 positioned relativeto a lung 1226.

For another example, as shown in one embodiment in FIG. 52 ,intraoperative C-arm CT imaging can cooperate with flexible scopeimaging to track a tumor or other critical structure for improvedcontrol in hard to access locations.

In some embodiments, controlling cooperative surgical imaginginteractions includes using smart device location cooperatively withscope tracking. In general, a non-magnetic sensing system can be usedfor 3D tracking to provide X, Y, Z coordinates using a single receiverand at least one emitter. A time-of-flight distance sensor system,discussed above, may thus be used.

For example, the non-magnetic sensing system can include ultrasonicsensor technology and radiofrequency (RF) sensor technology. A time offlight system can include an emitter and a receiver. To facilitatecontrolling cooperative surgical imaging interactions, the emitterincludes an ultrasonic sensor (ultrasonic beacon) configured to transmitultrasonic pulses, and the receiver includes an RF receiver configuredto transmit an RF signal that commands the emitter to begin transmittingthe ultrasonic pulses. The ultrasonic pulses are reflected back byobject(s) within their range. The RF receiver is configured to recordthe ultrasonic pulses and to, based on the recorded ultrasonic pulses,calculate 3D coordinates (X, Y, Z) of the emitter. The sound propagationtime of the ultrasonic pulses allows the RF receiver to calculate the 3Dcoordinates and to calculate distance to objects.

FIG. 53 illustrates one embodiment in which a camera 1260 includes areceiver 1262, a flexible scope 1264 includes a first ultrasonic sensor(obscured in FIG. 53 ), and a surgical instrument 1266 includes a secondultrasonic sensor (obscured in FIG. 53 ). The camera 1260 is positionedrelative to a lung outside of the lung. The scope 1264 has been advancedinto the lung and is positioned in the lung, and the instrument 1266 hasbeen advanced through a working channel of the scope 1264 to extenddistally out of the scope 1264. FIG. 53 shows the lung 1268 i in aninflated state (lung in dotted line) and shows the lung 1268 c in acollapsed state (lung in solid line), which illustrates a working space1270 made available after lung collapse for, e.g., tools such as thecamera 1260 to be effectively positioned in the patient. Arrows pointinginward in FIG. 53 indicate the lung collapse. 3D coordinates (X, Y, Z)can thus be calculated for the instrument 1266 and the scope 1264.

Controlling Power

Devices, systems, and methods for multi-source imaging provided hereinmay allow for controlling power. In general, controlling power canprevent an ablation device from applying too much power to tissue via anelectrode of the ablation device, which can result in inadvertentcollateral damage such as tissue wall perforation. The power output byan ablation device can be monitored, thereby allowing for the powercontrol. Examples of ablation devices are NeuWave™ ablation probes(available from Ethicon US LLC of Cincinnati, OH). Another example of anablation device is the Hyblate catheter ablation probe (available fromHyblate Medical of Misgav, Israel). Another example of an ablationdevice is the Barxx™ HaloFlex (available from Medtronic of Minneapolis,MN). Other examples of ablation devices are discussed further in, forexample, U.S. Pat. App No. 17/494,364 entitled “Surgical Methods ForControl Of One Visualization With Another” filed on Oct. 5, 2021.

After a laparoscopic portion of a segmentectomy (segment resection) hasbeen completed, an ablation device can be advanced through a workingchannel of a flexible endoscope and used to ablate a tumor or othertarget tissue. The ablation device can be a radiofrequency (RF) cauterytool or a microwave ablation tool. Ablation devices can be used toablate tissue in other types of surgical procedures as well.

The energy (power) of an ablation device can be controlled byvisualizing thermal or therapeutic effects from an opposite side of atissue wall from where the ablation device is applying the energy. FIG.54 illustrates one embodiment in which the energy (power) of a microwaveablation device (probe) 1280 can be controlled by visualizing thermal ortherapeutic effects from an opposite side of a tissue wall 1282, e.g.,using an infrared (IR) thermal camera 1284 positioned outside the tissuewall 1282, from where the ablation device 1280 is applying the energy toa tumor 1286. The tissue wall 1282 is a lung wall in this illustratedembodiment, but thermal or therapeutic effects can be similarlyvisualized and used in controlling power in surgical proceduresperformed elsewhere, such as in a duodenal mucosal resurfacing procedurein which duodenal mucosa is ablated. Infrared images 1288 gathered bythe IR thermal camera at four times t₀, t₂, t₄, t₆ are shown in FIG. 54.

FIG. 54 also shows a graph indicating each of IR camera 1284 temperature(°C), ablation probe 1280 position (cm), and ablation probe 1280 powerlevel (W) versus time. As indicated in the graph, the IR thermal camera1284 monitors the temperature of the ablation device’s ablation zone(e.g., an area of ablation that an ablation device 1280 can create),which includes the tumor 1286 and a margin area around the tumor 1286within the ablation zone. The times in the graph include the four timest₀, t₂, t₄, t₆ for which IR images are shown.

The therapeutic temperature range for tissue ablation is in a range fromabout 60° C. to about 100° C. As shown in the graph, the power level iscontrolled so the tumor 1286 is being ablated within the therapeutictemperature range from a time between times t₁ and t₂ to when energyapplication stops shortly before time t₆ between times t₅ and t₆. Thetemperature decreases from time t₂ and t₃, so the power level isincreased at time t₃ to prevent the temperature from falling below about60° C. The temperature increases from time t₃ and t₄, so the power levelis decreased at time t₄ to prevent the temperature from rising aboveabout 100° C. The temperature decreases again shortly before time t₅, sothe power level is increased at time t₅ to prevent the temperature fromfalling below about 60° C.

Tissue being at a temperature up to about 41° C. can cause blood vesseldilation and increased blood perfusion and trigger a heat-shock responsebut have little long term effect. The tissue being at a temperatureabove about 41° C. makes the tissue susceptible to or causes the tissueto incur irreversible cell damage, with the higher the temperaturecorresponding to more damage. As shown in the graph, the power level isalso controlled so the temperature of the healthy, non-target lungtissue in the ablation zone is kept below about 41° C. for as long aspossible while maintaining effective ablation of the tumor 1286.

For yet another example, an ablation device can include a plurality ofelectrodes that are equally spaced about a center of the device. Theplurality of electrodes are configured to move between a compressedconfiguration, which facilitates movement of the ablation device intoand out of a patient, and an expanded configuration, which facilitatesenergy application to tissue by the electrodes that each contacts thetissue. Each of the electrodes is segmented and is configured to takeinitial impedance and temperature readings. One embodiment of such anablation device 1290 is shown in FIG. 55 , FIG. 56 , and FIG. 57 . FIG.55 shows the ablation device 1290 in a compressed configuration in whichthe ablation device’s electrodes extend linearly. FIG. 56 and FIG. 57show the ablation device 1290 in an expanded configuration in which theelectrodes are radially expanded. FIG. 57 also shows the ablation device1290 advanced into position relative to a tumor 1292 through anendoscope 1294, with the electrodes advanced distally out of theablation device’s sheath 1290. The distal advancement of the electrodesout of the sheath 1296 (or proximal movement of the sheath 1296 relativeto the electrodes) causes the electrodes to automatically radiallyexpand. Correspondingly, proximal movement of the electrodes into thesheath 1296 (or distal movement of the sheath 1296 relative to theelectrodes) causes the electrodes to automatically radially contract. Insome embodiments, the sheath 1296 may be omitted such that movement ofthe electrodes into and out of the endoscope 1294 causes expansion andcompression of the electrodes.

To facilitate power control, each of the electrodes can contact tissueon one side of a tissue wall. A camera, e.g., the IR thermal camera 1284of FIG. 54 , can be positioned on an opposite side of the tissue wall.The electrodes can then begin applying a low level of power to thetissue. These low level energy pulses can be used to align the camera tothe precise area to be internally heated. The ablation device 1290 canshare the electrodes' initial impedance and temperature readings, whichcan serve as a baseline. The electrodes can then begin applyingtherapeutic energy, with the power being controlled based on the thermalenergy measured by the camera as discussed above, such as by adjustingat least one variable parameter of a control algorithm. Such an approachcan be used in a bipolar or a monopolar configuration. Introducingsaline between the mucosa and submucosa may help control the tissueheating to just the mucosa and to more specific depths by impedancemonitoring. The ablation device 1290 can be rotated clockwise orcounterclockwise such that the electrodes correspondingly rotateclockwise or counterclockwise, which may provide higher current densityin the event that additional heating is required, such as if the tissueis not being heated to the therapeutic ablation temperature range.

Color of tissue adjacent to tissue being ablated can be monitored andused to control power. The color of tissue can be used to indicateeither the heal of the tissue or the tissue’s vascularization capacity.In general, a color change over time can indicate that ablation energylevel should be adjusted.

FIG. 58 illustrates one embodiment in which tissue color can be used tocontrol power. A flexible scope 1295 has been advanced into a body lumen1296. An ablation device 1297 has been advanced through a workingchannel of the scope 1295 to extend distally beyond the scope 1295 suchthat an expandable member1297f of the ablation device 1297 has expandedradially. A laparoscope 1298 has been positioned outside the lumen 1296so as to be positioned on an opposite side of a tissue wall 1296 w(defined by the lumen 1296) than the scope 1295 and the ablation device1297. A field of view 1298 v is shown for the laparoscope 1298. Beforethe ablation device 1297 begins applying energy to the tissue via anelectrode (which can be one or more electrodes) of the ablation device1297, the laparoscope 1298 establishes an average color scale for tissuewithin its field of view 1298 w, within which energy will be applied onthe other side of the tissue wall 1296 w. Thereafter, as energy isapplied to the tissue via the electrode and the tissue is cooked fromthe inside, the tissue will tend to change color by becoming lighter.For example, the average color scale of tissue can be a pinkish red, andthe energy application can cause the tissue to lighten to a lighterpinkish red or to a whitish shade. As the tissue changes color duringenergy application, and the laparoscope 1298 continues to monitor thetissue color, the laparoscope 1298 can trigger the energy level tochange, e.g., providing a signal to a surgical hub, robotic surgicalsystem, or other computer system also in communication with the ablationdevice 1297 that triggers a change in at least one variable parameter,in response to the color reaching a predetermined light shade. There maybe a plurality of predetermined light shades, each corresponding to adifferent energy control change. The change in energy level may bemoving to an energy level of zero so as to stop energy delivery, as thetissue color change may indicate that the tissue is being coagulatedsuch that energy application should be stopped. Additionally, bubbling,effects of escaping steam, or effects of higher temperature fluids onadjacent tissue(s) can be monitored, either directly or by monitoringthe effects on the adjacent tissue(s).

In some embodiments in which tissue is ablated, a flexible force probecan be used to help keep a consistent thickness of tissue, which mayhelp ensure that the tissue eventually ablated has a consistentthickness. Consistent thickness generally helps ensure consistentheating of the tissue. FIG. 59 and FIG. 60 illustrate one embodiment ofa flexible force probe 1293 configured to apply force to a tissue wall1291 from one side of the tissue wall 1291, which in this illustratedembodiment is outside of a body lumen 1289 in which a scope 1287 ispositioned. The lumen 1289 is not shown in FIG. 60 . The flexible forceprobe 1293 includes an arm 1293 a configured to abut and press againstthe tissue wall 1291. The arm 1293 a can be flexible to allow the arm1293 a to conform to the shape of the tissue wall 1291. With the scope1287 positioned on the opposite side of the tissue wall 1291 while thearm 1293 a is pressing against the tissue wall 1291, a thickness 1291 tof the tissue wall 1291 can be controlled to a consistent thickness.Since the internal path of the scope 1287 is known within the lumen1289, the probe 1293 external to the lumen 1289 can follow the samespline path with the arm 1293 a pressing on the tissue wall 1291 tocontrol the tissue thickness 1291 t.

In some embodiments in which tissue is ablated using an ablation device,visualization can be used to monitor either collateral effects of theablation device’s energy application or the active intentional use ofthe ablation device near critical structure(s). The monitoring may helpprevent inadvertent damage. The monitoring can include monitoring alocation of the ablation device and data communicated from the ablationdevice indicating that the ablation device is active (e.g., is applyingenergy), monitoring tissue temperature, monitoring color of ejectedsteam, and/or monitoring effects caused by the ablation device. Themonitoring can serve a means to compare the ablation device’s affectedablation zone to surrounding critical structure(s) location andsensitivity. For example, multi-spectral imaging can be used to detect aureter in proximity to a location where the ablation device is applyingenergy and to trigger the ablating to stop if the ablation zone becomeswithin a threshold distance of the ureter.

Devices and systems disclosed herein can be designed to be disposed ofafter a single use, or they can be designed to be used multiple times.In either case, however, the devices can be reconditioned for reuseafter at least one use. Reconditioning can include any combination ofthe steps of disassembly of the devices, followed by cleaning orreplacement of particular pieces, and subsequent reassembly. Inparticular, the devices can be disassembled, and any number of theparticular pieces or parts of the device can be selectively replaced orremoved in any combination. Upon cleaning and/or replacement ofparticular parts, the devices can be reassembled for subsequent useeither at a reconditioning facility, or by a surgical team immediatelyprior to a surgical procedure. Those skilled in the art will appreciatethat reconditioning of a device can utilize a variety of techniques fordisassembly, cleaning/replacement, and reassembly. Use of suchtechniques, and the resulting reconditioned device, are all within thescope of the present application.

It can be preferred that devices disclosed herein be sterilized beforeuse. This can be done by any number of ways known to those skilled inthe art including beta or gamma radiation, ethylene oxide, steam, and aliquid bath (e.g., cold soak). An exemplary embodiment of sterilizing adevice including internal circuitry is described in more detail in U.S.Pat. No. 8,114,345 issued Feb. 14, 2012 and entitled “System And MethodOf Sterilizing An Implantable Medical Device.” It is preferred thatdevice, if implanted, is hermetically sealed. This can be done by anynumber of ways known to those skilled in the art.

The present disclosure has been described above by way of example onlywithin the context of the overall disclosure provided herein. It will beappreciated that modifications within the spirit and scope of the claimsmay be made without departing from the overall scope of the presentdisclosure. All publications and references cited herein are expresslyincorporated herein by reference in their entirety for all purposes.

What is claimed is:
 1. A surgical method, comprising: visualizing, witha first imaging system, a first side of a tissue wall during performanceof a surgical procedure, the first imaging system being a thoracoscopicimaging system or a laparoscopic imaging system; visualizing, with anendoscopic imaging system, a second, opposite side of the tissue wallduring the performance of the surgical procedure, the first imagingsystem and the endoscopic imaging system being unable to directlyvisualize each other, and the first imaging system and the endoscopicimaging system being operatively coupled together; guiding a firstsurgical instrument based on the visualization provided by the firstimaging system and the visualization provided by the endoscopic imagingsystem; and guiding a second surgical instrument configured based on thevisualization provided by the endoscopic imaging system and based on thevisualization provided by the first imaging system.
 2. The method ofclaim 1, wherein at least one of the first and second surgicalinstruments includes integrated tracking and coordinating means thatidentifies a location of the at least one first surgical instrument andsecond surgical instrument; and the method further comprisescommunicating the location to the first imaging system and theendoscopic imaging system so as to provide secondary locationverification to the first imaging system and the endoscopic imagingsystem.
 3. The method of claim 1, further comprising determining, withthe first imaging system, a location and orientation of the secondsurgical instrument relative to the first surgical instrument tofacilitate the first surgical instrument being guided based on thevisualization provided by the endoscopic imaging system; anddetermining, with the endoscopic imaging system, a location andorientation of the first surgical instrument relative to the secondsurgical instrument to facilitate the second surgical instrument beingguided based on the visualization provided by the first imaging system.4. The method of claim 1, further comprising visualizing, with each ofthe first imaging system and the endoscopic imaging system, a commonanatomic landmark and thereby facilitate guiding of the first and secondsurgical instruments relative to one another.
 5. The method of claim 1,wherein one of the first imaging system and the endoscopic imagingsystem includes a structured light emitter configured to emit astructured light pattern on a surface of an organ, a spectral lightemitter configured to emit spectral light in a plurality of wavelengthscapable of penetrating the organ and reaching an internal aspect locatedbelow the surface of the organ, and an image sensor configured to detectreflected structured light pattern, reflected spectral light, andreflected visible light; and the surgical method further comprisesconstructing, with a controller, a three-dimensional (3D) digitalrepresentation of the organ from the reflected structured light patterndetected by the image sensor, detecting, with the controller, theinternal aspect from the reflected spectral light detected by the imagesensor, integrating, with the controller, the internal aspect with the3D digital representation of the organ, and causing, with thecontroller, a display device to show the internal aspect and a 3Drendering of the organ based on the 3D digital representation of theorgan.
 6. The method of claim 1, further comprising visualizing, with acomputerized tomography (CT) imaging system, the first and secondinstruments and thereby facilitate guiding of the first and secondsurgical instruments relative to one another.
 7. The method of claim 1,further comprising gathering, with a non-magnetic sensing system thatuses at least one of ultrasonic energy and radiofrequency (RF) energy,data indicative of a location of the first and second surgicalinstruments and thereby facilitate guiding of the first and secondsurgical instruments relative to one another.
 8. The method of claim 1,wherein the first and second surgical instruments are each releasablycoupled to a robotic surgical system that controls the guiding of eachof the first and second surgical instruments.
 9. A surgical method,comprising: gathering, with an imaging device, images of an organ duringperformance of a surgical procedure, the images including a plurality ofimages each gathered at a different time during the performance of thesurgical procedure as the organ undergoes distortions; tracking, with acontroller, the organ from a first state to a second, different statethat is subsequent to the first state by analyzing the plurality ofimages, the tracking allowing the controller to predict an internalaspect of the organ in the second state, and causing, with thecontroller, a display device to show an indication of the internalaspect of the organ in the second state.
 10. The method of claim 9,wherein the tracking forecasts a perimeter shape of the internal aspectin the second state, a location of the internal aspect in the organ inthe second state, and an orientation of the internal aspect in the organin the second state.
 11. The method of claim 9, wherein the plurality ofimages are analyzed with respect to at least one of tissue strain and ageometry of the organ.
 12. The method of claim 9, wherein the imagingdevice includes a flexible scoping device, the organ is a lung, theinternal aspect is a tumor.
 13. The method of claim 12, wherein the lungin the first state is expanded, and the lung in the second state iscollapsed.
 14. The method of claim 13, further comprising controlling,with the controller, the collapse of the lung to create a predefinedshape of the tumor.
 15. The method of claim 9, wherein the imagingdevice includes a structured light emitter configured to emit astructured light pattern on a surface of the organ, a spectral lightemitter configured to emit spectral light in a plurality of wavelengthscapable of penetrating the organ and reaching the internal aspectlocated below the surface of the organ, and an image sensor configuredto detect reflected structured light pattern, reflected spectral light,and reflected visible light; and the method further comprisesconstructing, with a controller, a three-dimensional (3D) digitalrepresentation of the organ from the reflected structured light patterndetected by the image sensor, detecting, with the controller, theinternal aspect from the reflected spectral light detected by the imagesensor, integrating, with the controller, the internal aspect with the3D digital representation of the organ, and causing, with thecontroller, a display device to show the internal aspect and a 3Drendering of the organ based on the 3D digital representation of theorgan.
 16. The method of claim 9, wherein the organ is a lung; theimaging device includes a bronchoscope; the method further comprisesadvancing the bronchoscope distally in an airway of the lung during theperformance of the surgical procedure; and the method further comprisesautomatically causing, with the controller, the bronchoscope to beretracted proximally in the airway of the lung during the performance ofthe surgical procedure.
 17. The method of claim 16, further comprisingautomatically causing, with the controller, the bronchoscope to beretracted proximally based on an analysis of the images indicating thebronchoscope being within a threshold distance of a tissue wall.
 18. Themethod of claim 17, further comprising automatically causing, with thecontroller, the bronchoscope to be retracted proximally also based on atleast one of a rate of change of a volume of the lung and a rate ofchange of an external surface of the lung.
 19. The method of claim 18,further comprising automatically causing, with the controller, thebronchoscope to be retracted proximally also based on the rate of changeof the volume of the lung as indicated by at least one of CT imaging andventilator exchange.
 20. The method of claim 18, wherein the imagingdevice includes a structured light emitter that emits a structured lightpattern on the external surface of the organ; the imaging deviceincludes an image sensor that detects reflected structured lightpattern; and the method further comprises automatically causing, withthe controller, the bronchoscope to be retracted proximally also basedon the rate of change of the external surface of the lung as indicatedby the reflected structure light pattern.
 21. The method of claim 9,wherein a surgical hub includes the controller.
 22. The method of claim9, wherein a robotic surgical system includes the controller; and theimaging device is releasably coupled to and controlled by the roboticsurgical system.